Skip to content
Working with the tools, not about them

Pitfalls

A research answer arrives already sorted, and the sorting is the problem

Asking what is known about a subject returns a settled-sounding account, and what the settling removed is exactly what a researcher needed to see.

By Aarav Sinha3 min read

Editorial note. Independent reporting and analysis. Nothing here is sponsored or paid for. How we work.

Consensus is manufactured by the request

Ask what the evidence says about a question and the answer will describe a position, note that some disagreement exists, and move on. It reads like a literature review. It is closer to an average, and an average of a contested field is a position that nobody in the field holds.

The specific loss is the shape of the disagreement: who disputes what, on what grounds, and whether the dispute is about the data, the method or the interpretation. That structure is the useful part of knowing a field, and it is precisely what gets compressed away when a question asks for what is known.

The second loss is chronology. Findings that were superseded, positions that shifted after a particular result, and the fact that a question was live five years ago and is settled now, or the reverse, tend to flatten into a single timeless present tense.

References are the most dangerous part of the output

A citation looks like the most checkable element in a research answer, which is why it is trusted disproportionately. It is also the element most likely to be constructed: a plausible author, a plausible title, a plausible journal, in the right format, describing a paper that doesn’t exist or that says something else.

Where a tool is genuinely retrieving documents rather than composing a reference, this is less severe, and the failure changes shape rather than disappearing. Real sources get cited for claims they do not support, or a supporting sentence is quoted without the qualification in the following paragraph.

The rule that follows is unambiguous and unpopular: no citation goes into anything you publish or submit until you have opened it and read the relevant part. Not the abstract. The part that supposedly supports the claim.

This is also why an answer with no citations is sometimes safer than one with a neat list. The list creates an impression of grounding that the reader has no way to test, and it will be trusted by everyone downstream who assumes somebody checked.

Use it to map, then read the map yourself

The reliable contribution to research is orientation. What is this area called, what are the competing schools, which terms would a specialist search for, what are the standard objections, what would I need to read first. These outputs are checkable within minutes and they save hours of vocabulary-hunting.

That is a genuinely large saving for anyone entering an unfamiliar field, and it is worth separating from the unreliable part rather than dismissing both together. Names of things are cheap to verify. Claims about what the evidence shows aren’t.

The discipline is to treat every output as a pointer rather than a finding. If a claim matters, it survives only after you have found the primary source and read enough of it to see the conditions attached.

The gaps are the ones the field does not talk about

A research summary reflects what is written about, and coverage across any field is uneven. Well-studied populations, well-funded questions, results that were published because they were positive, and work in widely read languages are all overrepresented relative to their importance.

So the absence of something in an answer is very weak evidence that it doesn’t exist. Practitioners often notice this first when a subject they know well comes back with the standard account and none of the exceptions that dominate their working life.

Asking directly what is understudied, or what a critic of this account would say, produces a more useful second half than any amount of asking for more detail. It targets the omissions rather than expanding the centre.

Where this belongs, and where it does not

For orientation, terminology and generating questions to take to a specialist, it is excellent and the cost of being wrong is small. For a literature review, a regulatory determination, a medical question, a legal position or anything that will be relied upon, it is a starting point and nothing more.

The distinction is not about difficulty; it is about who bears the consequence of an error. Work that only informs your own thinking can tolerate a wrong pointer, because you will meet the correction as you read. Work handed to someone else carries the error into a place where nobody is looking for it.

Stating which of the two you are producing, at the top of your own notes, is a small habit that resolves most of this. It is remarkably easy to begin a research task in the first mode and finish by pasting the result into something that lives in the second.

Common questions

Why do research answers sound more settled than the field is?

Because a question asking what is known invites a single account, and producing one means averaging over disagreement. The structure of the dispute — who objects, on what grounds, about data or method or interpretation — is exactly what the averaging removes.

Can I trust the citations?

Not without opening them. References are the element most likely to be constructed, and even where real documents are retrieved they are frequently cited for claims they do not support. Read the specific passage that supposedly supports the claim, not the abstract.

What is research help genuinely good for?

Orientation: the name of the field, the competing positions, the terms a specialist would search for, the standard objections and what to read first. Those are verifiable in minutes and save hours, unlike claims about what the evidence shows.

Pitfallsresearchsourcesevidenceverification
Aarav Sinha
Features writer, Prompt After Prompt

Aarav covers prompt craft, writing with ai, images & audio and the questions readers actually send in and thinks most subjects are more interesting once you know how they work.