Recursive retrieval-augmented composition for language models on open-ended questions
Collaborator(s): Justin Blumencranz. Mentors: Yucheng Jiang, Monica Lam
Deep research systems turn retrieval-augmented generation into report writing. A language model searches the web over many rounds and then writes a report that cites a source for each statement. OpenAI Deep Research, released in February 2025, and STORM, introduced by the Stanford OVAL lab in 2024, are two of the most widely used.
These systems are designed to produce surveys. For example, when asked whether the Federal Reserve should keep interest rates high, they present the evidence on each side and leave the judgment to the reader. While a survey is useful for preliminary research on a topic, we believe that many jobs are ultimately about reaching a decision and defending it. A lawyer writing a brief must argue for one side of a case, and an analyst advising on an investment must recommend whether to buy or sell. In both cases the reader expects a clear position, with evidence behind each claim that supports it, so a balanced summary leaves most of the work undone.
An argument also has a shape before any searching starts. The main claim rests on a few supporting claims, and each of those rests on evidence. A research system can plan this shape first, breaking the question into sub-questions small enough to answer by search and then combining the answers upward into a report that argues one side.
Planning. The pipeline runs in three phases (Figure 1). A short literature search on the root question first gives a planning model context. The model then decomposes the question layer by layer into a directed acyclic graph of sub-questions, conditioning each node's children on the structure of the whole graph so that siblings do not overlap and together answer their parent. A second pass writes, for every internal node, instructions for how the answers of its children combine into its own answer.
Recursive composition. The graph is processed from the leaves to the root. Each leaf is answered by web search, with retrieved passages ranked by embedding similarity to the question and every claim cited. Each parent is then composed only from its children's answers, under its composition instructions, and must state when its children contradict each other. A claim without a source therefore has little path into the final report.
Refinement. A composed answer can still miss something the plan did not anticipate. After composing a parent, a gap explorer lists what the answer leaves out, and a gap evaluator decides whether any omission matters. If one does, the pipeline attaches a new sub-tree of questions under that parent, answers it, and composes the parent again (Figure 2).
Report writing. An outline model turns the answered graph into sections that follow its branches, and a report model writes from that outline. The report is titled with the claim at the root rather than the question and keeps every citation from the graph. The number of model calls grows linearly with the size of the graph, about 40 for the run in Figure 2 against one or a few for a report written in a single pass.
Human evaluation. Each system wrote one report for each of ten contested questions on topics from ethics to technology, and one rater scored all thirty reports with the system hidden. De(ep)Composition achieves a supported-claims rate of 0.80, against 0.68 for OpenAI Deep Research and 0.60 for STORM. Its reports put a larger share of their cited sources to use in the argument, 0.76 against 0.67 and 0.61, and receive the highest score for logical connection, 7.7 of 10 against 6.5 and 6.0. It holds a firm stance in 9 of 10 reports, against 1 and 2.
Automated evaluation. In six automated adversarial exchanges with Deep Research on three further questions, each system argued both sides from its own research, and GPT-4o-mini scored the transcripts. De(ep)Composition holds its assigned side in 91% of turns, against 68%, and scores higher on every rubric measure except fluency, where its denser citation of sub-claims costs some flow (7.7 against 8.6).
Stance. Word counts on the ten retained pairs of De(ep)Composition and STORM reports support the stance result without a rater. De(ep)Composition uses fewer hedges such as may and could on 9 of 10 questions and more committing words such as must and therefore on all 10. Both systems acknowledge the other side at the same rate, measured by contrast markers such as however, so De(ep)Composition differs in committing after it does so. Its conclusions are softer than its titles, however. Only 5 of the 10 reports end as firmly as they begin.
Length and sourcing. De(ep)Composition's reports are shorter, a median of 1,339 words against 2,426 for STORM, and cite more densely, 5.7 citations per 100 words against 4.7 (Figure 3). They draw on far fewer distinct sources, a median of 8 against 27, so each source is cited about nine times. Each report retrieves a median of 36 sources, and the body cites about a quarter of them.
Citation fidelity. A citation can be present and still point to the wrong source. Ranking every source by word overlap with a cited sentence shows whether the cited source is the one the sentence came from. For citations known to be correct, the cited source ranks first for 96% of claims. In the ten evaluated reports it ranks first for 14%, close to the 10% expected from random numbers (Figure 4). The cause lies in how answers are combined. Each sub-question numbers its own sources from 1, and combining two answers mixed up their numbers. Giving every source one number for the whole run fixes the mix-up, and the cited source again ranks first for 96% of claims when the answers are combined.
Justin and I also put our marketing and video-editing skills to the test with this fun video!