When a user asks "What are the trade-offs between React and Vue for a large enterprise team with legacy jQuery code?", a single embedding of that sentence lands somewhere in the middle of the semantic space between React docs, Vue docs, enterprise architecture patterns, and jQuery migration guides. Cosine similarity against any individual chunk topic is diluted. The top-k results come back as a blurry mix that partially addresses each concern but fully addresses none. Query decomposition fixes this by acknowledging that a single embedding is a lossy compression of intent.
The mental model is simple: treat query decomposition as a planning step. Before touching the vector store, you ask an LLM to produce an explicit execution plan -- a list of retrievable atomic questions. "Atomic" means each sub-query can be answered by a single contiguous passage. A question like "how does React handle state management in large applications" is atomic. "Compare React and Vue" is not -- it requires two independent sets of evidence before any comparison is possible. Your decomposition prompt should enforce this. The LLM is your query planner; the vector store is your storage engine. Keep those roles separate.
In practice, a mid-size e-commerce company building an internal knowledge base over their product catalog, supplier contracts, and operations docs would immediately benefit here. A support agent asks: "Which suppliers offer next-day delivery for SKUs under $10, and what are the return policy differences between them?" That decomposes into: (1) which suppliers offer next-day delivery, (2) which SKUs are priced under $10, (3) what is each supplier's return policy. Each sub-query retrieves a tight set of chunks. A senior engineer would run those three retrievals concurrently using asyncio (covered in aidev-python-async), collect the unique chunks, deduplicate by chunk ID, then pass the merged context to a synthesis call. Total latency is dominated by the slowest single retrieval, not the sum of all retrievals.
Compared to alternatives: HyDE (hypothetical document embeddings) generates a fake answer and embeds that to improve retrieval -- it works well for single-concept queries but doesn't solve the multi-concept coverage problem. Step-back prompting asks the LLM to restate the query at a higher abstraction level before retrieving, which helps with specificity issues but still sends one query. Multi-query retrieval (used in LangChain's MultiQueryRetriever) generates paraphrases of the original query rather than truly decomposed sub-questions -- useful for recall but wasteful when the concepts are genuinely distinct. Decomposition is the right tool specifically when a query spans multiple separable knowledge domains. When the query is complex but single-domain ("explain how Kubernetes handles pod eviction during resource pressure"), decomposition adds overhead without much benefit.
At scale, decomposition introduces three concrete costs you need to plan for. First, LLM calls: every query now requires at least two LLM calls (decompose + synthesize) instead of one. At 10 users, irrelevant. At 10k daily active users, you're looking at a meaningful cost multiplier. Use the smallest capable model for decomposition -- gpt-4o-mini or claude-haiku does this well at a fraction of the cost of frontier models. Second, vector store read load: 3-4 sub-queries per original query multiplies your retrieval requests proportionally. At 10M queries per month, that matters for your Pinecone or Qdrant tier. Cache decomposition results for identical or near-identical queries using a lightweight semantic cache (embed the original query, check cosine similarity against recent decompositions). Third, context window pressure: N sub-queries each return top-k chunks. If N=4 and k=5, you have up to 20 chunks going into synthesis. Chunk deduplication by content hash and a re-ranking step (see aidev-advanced-rag-reranking) before synthesis keeps the context window manageable and improves answer quality by surfacing the most relevant chunks across all sub-queries.
Key Takeaways
- Decompose multi-faceted queries into atomic sub-queries before retrieval to avoid semantic dilution.
- Use a fast, cheap LLM call for decomposition -- it doesn't need to be your most powerful model.
- Run sub-query retrievals in parallel to avoid linear latency stacking.
- Deduplicate retrieved chunks across sub-queries before synthesis to reduce prompt bloat.
Pro tips
- Constrain your decomposition prompt to produce exactly 2-4 sub-questions and reject outputs outside that range. Unbounded decomposition causes prompt bloat and cost spikes on edge-case queries with no corresponding quality gain.
- Log the decomposed sub-queries in your observability pipeline alongside the original query. When users report wrong answers, the decomposition is almost always where the failure originates -- seeing what the model split the query into is the fastest debugging lever you have.
- Use a content hash of each retrieved chunk as the deduplication key rather than chunk text equality. The same chunk can be returned by multiple sub-queries with minor whitespace differences, and text equality checks will miss those duplicates.
- For queries that appear to be single-concept (short, no conjunctions, no comparative language), skip decomposition entirely. A lightweight classifier or even a simple heuristic like token count under 12 can gate whether you run the decomposition call, cutting unnecessary LLM spend by a meaningful amount in production.
Common pitfalls
- Mistake: Letting the LLM generate sub-queries that are interdependent ("What is X" then "How does X compare to Y"). Fix: Instruct the model explicitly that each sub-question must be answerable independently with no reference to other sub-questions.
- Mistake: Running sub-query retrievals sequentially instead of in parallel. Fix: Use asyncio.gather or a thread pool to run all retrievals concurrently; sequential execution stacks latency and is the most common performance regression in decomposition pipelines.
- Mistake: Skipping deduplication before synthesis, passing duplicate chunks to the final LLM call. Fix: Deduplicate by chunk ID or content hash before building the synthesis context; duplicates waste tokens and can bias the model toward over-cited facts.
- Mistake: Using your most powerful model for decomposition. Fix: Decomposition is a structured extraction task, not a reasoning task. A small, fast model like gpt-4o-mini handles it well at a fraction of the cost of GPT-4o or Claude Opus.
When to use query decomposition vs alternative query expansion techniques
| Option | Use when | Avoid when |
|---|---|---|
| Query decomposition | Query spans 2+ genuinely distinct knowledge domains or requires comparative evidence across separate topics. | Query is single-concept or short; the overhead of an extra LLM call isn't justified. |
| Multi-query retrieval (paraphrasing) | Query is single-concept but ambiguous; you want higher recall by covering synonyms and rephrasings. | Query topics are truly distinct -- paraphrases won't broaden coverage across separate domains. |
| HyDE (hypothetical document embedding) | Query is specific and well-formed but domain vocabulary in the index doesn't match the user's vocabulary. | Multi-faceted queries where the generated hypothesis will average across too many concepts. |
| Step-back prompting | Query is too narrow or overly specific, causing retrieval to miss higher-level explanatory content. | Query is already at the right abstraction level; stepping back loses the specificity the user needs. |
Code Example
# openai>=1.0.0, requires OPENAI_API_KEY env var
import os, json
from openai import OpenAI
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
def decompose_query(user_query: str) -> list[str]:
response = client.chat.completions.create(
model="gpt-4o-mini",
temperature=0,
messages=[
{
"role": "system",
"content": (
"Break the user question into 2-4 focused sub-questions, each "
"answerable from a single document passage. Return a JSON array of strings."
),
},
{"role": "user", "content": user_query},
],
response_format={"type": "json_object"},
)
data = json.loads(response.choices[0].message.content)
return data.get("sub_questions", [])
query = "Compare solar panel efficiency across climates and typical installation costs."
print(decompose_query(query))How this code works
This code efficiently breaks down a complex user question into multiple simpler, more focused sub-questions. Its purpose within an advanced RAG lesson is to facilitate query decomposition, making it easier for a system to retrieve precise answers. By transforming a broad query like "Compare solar panel efficiency across climates and typical installation costs" into several distinct sub-queries, the system can search for specific information about each aspect individually, leading to more accurate and comprehensive results.
The process begins by initializing an OpenAI client, which connects to OpenAI's powerful language models. The core logic resides in the decompose_query function. This function uses client.chat.completions.create to send the user_query to the gpt-4o-mini model. Crucially, a "system" message provides clear instructions, telling the AI to generate "2-4 focused sub-questions" and to return them as a "JSON array of strings." The response_format={"type": "json_object"} ensures the AI's output is consistently structured as JSON. An important subtle choice here is temperature=0, which makes the AI's responses highly deterministic; for query decomposition, consistent and predictable sub-questions are preferred over creative variations. Finally, json.loads converts the AI's JSON string into a Python object, and data.get("sub_questions", []) safely extracts the sub-questions, providing a robust way to handle potential variations in the AI's response format.
Production-grade example
Adds retries with backoff, timeouts, token logging, graceful degradation, parallel retrieval, and chunk deduplication.
# openai>=1.0.0, asyncio, structlog -- requires OPENAI_API_KEY env var
import asyncio, hashlib, json, logging, os, time
from typing import Any
import structlog
from openai import AsyncOpenAI, RateLimitError, APITimeoutError
log = structlog.get_logger()
client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"], timeout=10.0)
async def decompose_query(user_query: str, attempt: int = 0) -> list[str]:
max_retries = 3
try:
t0 = time.monotonic()
resp = await client.chat.completions.create(
model="gpt-4o-mini",
temperature=0,
max_tokens=300,
messages=[
{
"role": "system",
"content": (
"Decompose the question into 2-4 atomic sub-questions, "
"each answerable from a single document passage. "
'Return JSON: {"sub_questions": [...]}.'
),
},
{"role": "user", "content": user_query},
],
response_format={"type": "json_object"},
)
latency_ms = (time.monotonic() - t0) * 1000
usage = resp.usage
log.info("decompose_query", latency_ms=round(latency_ms),
prompt_tokens=usage.prompt_tokens,
completion_tokens=usage.completion_tokens)
data = json.loads(resp.choices[0].message.content)
return data.get("sub_questions", [user_query])
except (RateLimitError, APITimeoutError) as exc:
if attempt >= max_retries:
log.warning("decompose_query.fallback", error=str(exc))
return [user_query] # graceful degradation: treat original as single sub-query
backoff = 2 ** attempt
log.warning("decompose_query.retry", attempt=attempt, backoff=backoff)
await asyncio.sleep(backoff)
return await decompose_query(user_query, attempt + 1)
async def retrieve(sub_query: str) -> list[dict[str, Any]]:
# Stub: replace with your actual vector store call (Pinecone, Qdrant, etc.)
await asyncio.sleep(0.05)
return [{"chunk_id": hashlib.md5(sub_query.encode()).hexdigest(), "text": f"Stub result for: {sub_query}"}]
async def decompose_and_retrieve(user_query: str) -> dict[str, Any]:
sub_queries = await decompose_query(user_query)
results = await asyncio.gather(*[retrieve(q) for q in sub_queries])
seen, merged = set(), []
for chunks in results:
for chunk in chunks:
if chunk["chunk_id"] not in seen:
seen.add(chunk["chunk_id"])
merged.append(chunk)
log.info("retrieve.merged", sub_query_count=len(sub_queries), unique_chunks=len(merged))
return {"sub_queries": sub_queries, "chunks": merged}How this code works
This code demonstrates how to decompose a complex user query into smaller, more manageable sub-queries, then retrieve information for each. The decompose_and_retrieve function is the main entry point, first using decompose_query to break down a user_query into atomic sub_queries. It then concurrently calls retrieve for each sub-query using asyncio.gather to fetch relevant information, and finally merges these results, de-duplicating chunks by their chunk_id before returning the sub_queries and unique chunks.
The decompose_query function interacts with an AsyncOpenAI gpt-4o-mini model, sending the user_query and a system prompt to generate a JSON list of sub_questions. A subtle but critical feature is its retry logic: if it encounters RateLimitError or APITimeoutError, it won with exponential backoff (asyncio.sleep) for up to max_retries. If all retries fail, it gracefully degrades by treating the original user_query as a single sub-query, preventing a hard crash. The retrieve function is a placeholder that would be replaced with an actual call to a vector store.
Practice & master
Try the exercise, check your understanding, then mark this lesson mastered to track your path to pro.
Exercise
Build a decompose-then-retrieve pipeline that takes the query "What are the latency and cost differences between GPT-4o and Claude 3.5 Sonnet, and which is better for real-time chatbots?" Decompose it into sub-queries, run mock retrievals in parallel using asyncio, deduplicate the results, and print the merged chunks.
import asyncio, json, os
from openai import AsyncOpenAI
client = AsyncOpenAI(api_key=os.environ["OPENAI_API_KEY"])
MOCK_CHUNKS = {
"latency": [{"id": "c1", "text": "GPT-4o median latency is ~500ms"}],
"cost": [{"id": "c2", "text": "Claude 3.5 Sonnet costs less per output token"}],
"chatbot": [{"id": "c1", "text": "GPT-4o median latency is ~500ms"}, {"id": "c3", "text": "Streaming support matters for real-time UX"}],
}
async def decompose(query: str) -> list[str]:
# TODO: call gpt-4o-mini with a decomposition system prompt
# return a list of sub-question strings
pass
async def mock_retrieve(sub_query: str) -> list[dict]:
# TODO: pick the right MOCK_CHUNKS key based on keywords in sub_query
# fall back to [] if no match
pass
async def main():
query = "What are the latency and cost differences between GPT-4o and Claude 3.5 Sonnet, and which is better for real-time chatbots?"
sub_queries = await decompose(query)
print("Sub-queries:", sub_queries)
# TODO: run mock_retrieve for all sub_queries in parallel
# TODO: deduplicate merged results by chunk id
# TODO: print final merged chunks
asyncio.run(main())Quick check
Why does embedding a long multi-concept query as a single vector hurt retrieval quality?
Which execution pattern keeps total latency close to the slowest single sub-query retrieval rather than their sum?
A user query is 8 tokens and asks a single narrow question. Should you apply decomposition?