Keep a research agent's output as a draft until you have checked the claims you intend to publish. A useful review needs more than a list of links. Each important statement needs a source that supports its wording, and each dataset needs an honest account of what the search covered.
This guide proposes an editorial review procedure. The two accounts below document actual research runs, but we have not reproduced them. Their results illustrate why citation support and search coverage deserve separate checks. The review template and stopping rules are our recommendations, rather than procedures claimed by the authors.
Preserve the run before you edit
Save the original request, returned artifact and available tool trace together. Record the run date and the intended audience. Keep that copy separate from your edited version so you can distinguish the agent's claims from your corrections.
Simon Willison's orchestrion research account links a returned JSON dataset and a view of the research trace. He describes the list as incomplete and its contents as insufficiently verified for publication beyond his account of the experiment. Our directory entry preserves that limitation.
Start your own review with the same distinction between a record of a run and a reliable reference work. A completed run tells you that an artifact exists. It does not tell you that the artifact answers the question fully.
Make each claim independently reviewable
Break compound statements into claims that a source can support individually. An entry saying that a museum owns an instrument, displays it and offers public demonstrations contains separate assertions. A historical collection catalogue might support ownership while saying nothing about current visitor access.
For each claim, keep the exact proposed wording beside the source URL and the relevant passage or page number. Add the source date when available and the date you accessed it. If a page has changed, preserve enough bibliographic detail for another reviewer to identify the version you used.
Use a small review record like this:
Claim ID: venue-04-access
Proposed claim: [one factual statement]
Primary source: [URL and page or section]
Supporting passage: [short excerpt or precise location]
Source date: [date, or unknown]
Checked on: [date]
Decision: supported / revise / unresolved / contradicted
Reviewer note: [reason and any narrower wording]
Keep this record with the dataset. A future editor can then update the affected claim without repeating every search. Avoid treating an entire row as verified when only one field has supporting evidence.
Check support separately from citation presence
Open the cited source and compare its wording with the claim. Check who performed the work, which population or object it concerns, and whether the statement describes a proposal or an observed result. Preserve qualifications that affect the meaning.
FutureHouse's WikiCrow report describes generating draft gene summaries using its PaperQA agent. Its evaluation asks separately whether statements have relevant citations and whether those citations support the statements. The authors also report incorrect statements and ask readers to check information before relying on it. These are draft summaries, not evidence of accepted Wikipedia publication. See our WikiCrow entry for the deployment summary.
Apply that separation to your draft. A relevant paper can discuss the right gene while failing to support the claimed function. Mark the claim unresolved if you cannot inspect the supporting material. Ask the agent to locate accessible evidence or narrow the statement, then review the revision against that evidence.
Resolve identities before merging records
Check that repeated names refer to the same entity. Record aliases separately from a stable identifier where one exists. For a venue, compare its address and official website. For a publication, compare its title, authors and publication identifier.
Define what one row represents before removing duplicates. A directory of venues and a catalogue of instruments need different rules. Several instruments at one museum can be separate catalogue records while sharing a single venue record. Merging by address alone would erase that distinction.
Treat coordinates as claims too. Check the coordinate order and inspect the location on a map. A point can be syntactically valid while identifying a city centre rather than the named building. Retain an uncertainty note when the source provides only an approximate location.
Describe coverage without claiming completeness
Write down the inclusion criteria before deciding that the search is finished. Specify the geography, date range, languages and source types you intend to cover. Keep a record of inaccessible sources and ambiguous candidates.
Test coverage with a few known examples chosen independently of the returned list. If the agent misses an example that meets your criteria, investigate why. It may have searched the wrong terminology, excluded a source format or combined distinct entities. Correct the search procedure before interpreting the result count.
Passing these spot checks still does not establish completeness. Publish the search scope and known gaps alongside the artifact. When you lack an authoritative inventory for comparison, describe the result as a collection of located examples rather than every qualifying case.
Publish only the reviewed claims
Review every claim that determines inclusion, especially identity and current status. For less consequential descriptive fields, document any sampling and leave unchecked fields visibly marked. Do not let a successful sample silently certify the whole dataset.
Keep unresolved rows in a separate working file when readers would otherwise mistake them for confirmed records. If you publish a provisional dataset, state which fields remain unchecked and what the artifact is suitable for. A list of research leads needs different wording from a visitor guide.
Finish with a dated review note that records the scope, unresolved questions and correction route. Preserve the evidence records alongside the published version. When a source changes or a reader challenges a claim, use those records to revise the affected statement and explain the correction.