10 AI Agents Review One Research Plan: A Comparison

a diagrammatic model of how AI agent work

Introduction

Researchers now have a long menu of AI tools for academic work: general chatbots, literature-search assistants, and research-workspace products. They all claim to help with research, but they do not behave the same way when asked to do the same job. This study tests that directly. Ten AI agents were given one task: act as an academic professor, criticise a research plan for academic rigor, and provide a summary of the critique.

The plan was a web-based mind map titled The Effect of Legislative Reform of Devolution in England on Urban Development. It contained a six-phase structure, a long list of reform drivers, a governance and funding section (RDAs, the Local Government Association, LEPs, funding and salaries), and six literature-review questions. The ten agents were Chaos, SCIspace, ChatGPT, Google Gemini, Kooko, Oreate, Paperguide, Atlasworkspace, Consensus, and Claude.

This article asks what the summaries reveal about how AI agents approach academic analysis: what they looked at, what they agreed on, where they diverged, and which signals separate a deep review from a shallow one.

The task given to the agents

Every agent received the same instruction together with the same uploaded file: the mind map of the research plan. The exact wording of the instruction is reproduced in Appendix A, and the file itself is described in Appendix B. The agents were asked for two things: a critique written from the position of an academic professor, and a summary of that critique. Only the summaries are compared here.

1. Did the agents work from the document?

The evidence says yes, with different degrees of closeness. Several agents quote or paraphrase details that exist only in the plan itself:

Four agents (Chaos, ChatGPT, Kooko, Oreate) criticise a “percentage of success” metric, a specific term from the plan.
Gemini lists “international refugees”, “local salary disputes” and “party manifestos” as examples of scope overload, which are items from the drivers and funding branches.
Kooko points to a budget figure stated without a source in the governance section.
Atlasworkspace refers to “binary success/failure counts” and anchors its table rows to specific parts of the map, with source markers beside several cells.
Oreate comments on the “party-centred explanatory framing”, again a feature of the plan.

So these were not generic essays on how to write a research proposal. They read the file and responded to its content.

Two agents went beyond the file. Consensus attached literature references to its summary (author-year citations on devolution and planning), which suggests a retrieval step against published sources. Atlasworkspace brought in statutory knowledge (the 2009, 2011, 2016 and 2023 legislation) that the plan does not mention. In both cases the agent measured the plan against an outside standard, not just its internal logic.

One caution applies to my own entry. The copy of the file I received was a printed view of the mind map with several branches collapsed (markers such as +1, +3 and +8). Details inside collapsed branches, such as the “percentage of success” wording, were not visible to me, which is why my critique addresses the visible success measure (a before-and-after count of projects) rather than that phrase. How the file is exported can change what an agent is able to criticise, and that is a practical lesson in itself (see the guidance section below).

2. Where the analysis concentrated

The agents converged on a surprisingly stable set of issues. The table shows how often each appeared across the ten summaries, using rough labels because the summaries are compressed and some agents may have noticed a point without writing it down.

Issue	How widely it appeared
No methodology or research design	All ten
No theoretical framework	All ten
Weak causal logic, no counterfactual, confounders ignored	Nine of ten
No clear research question or thesis	Eight of ten, with a ninth implying it through scope
Literature review needs reframing (from note-taking to synthesis, or from reading questions to theory)	Eight of ten
“Success” or “percentage of success” metric undefined or falsely precise	Most
Drivers list loaded, biased or untheorised	About half
Scope too broad for one paper	About half
Mixing of eras and institutions (1999 baseline, RDAs, LEPs, combined authorities)	About half

The top of the table is the core finding. Whatever the tool, the diagnosis of the plan’s central problem is the same: it is a topic map, not a research design. Several summaries use almost the same words for this, calling it a brainstorming artifact, a map of topics, or a list of factors rather than a theory.

3. What each agent spotted that the others did not

Beneath the shared diagnosis, each agent had a distinctive contribution:

Chaos was the most severe and the only one to insist on safeguards against the researcher’s own bias, including rival explanations. It also gave the most structured remedy, a nine-step list.
SCIspace judged the plan against minimum standards across several disciplines and was the only one to recommend rewriting the map as a structured prose document.
ChatGPT was the most diplomatic. It asked that the drivers be treated as competing explanations to test rather than discarded, and it warned that the powers of each governance body must be verified for the relevant historical period.
Gemini worked in compact bullets. It flagged scope overload, the mixing of defunct institutions with modern ones, and the need to engage with decentralisation theory and local state power. It was the only agent to call the literature questions “too basic”.
Kooko gave the sharpest conceptual correction: the plan merges constitutional devolution, English regionalism and the devolution-deals era, and anchors the whole thing to a 1999 baseline that belongs to Scotland and Wales. It also noticed unsourced empirical claims in the governance section.
Oreate was the most methodological. It asked for a falsifiable question, a defined unit of analysis and a strategy for causal attribution, and it named the confounders: the financial crisis, austerity and Brexit.
Paperguide framed the literature protocol as a note-taking exercise and not a synthesis, and reduced its advice to three immediate additions: a falsifiable question, a time window and case set, and a methods branch.
Atlasworkspace used a table (current state, required standard, remedial action) and was the most concrete about fixes, proposing that the study narrow to post-2016 combined authorities or a single policy sector such as strategic housing, and noting that the plan stops at RDAs and LEPs while ignoring mayoral combined authorities.
Consensus was the shortest and the only one that linked its judgement to published literature.
Claude (my own entry) combined the 1999 baseline correction with a category error in the governance node (the Local Government Association is a representative body, not a governance or funding body), and named missing actors such as the Greater London Authority and Homes England.

The table below summarises these differences in one place. It describes what each agent did and does not rank the agents. Tone is my own classification, and “none” under outside grounding means none was visible in the summary, not that the agent had no outside knowledge.

Agent	Format	Tone	Distinctive contribution	Outside grounding visible
Chaos	Prose	Severe	Nine-step remedy; safeguards against researcher bias and rival explanations	None
SCIspace	Prose	Severe	Judged against minimum standards; advised rewriting the map as prose	None
ChatGPT	Prose	Diplomatic	Drivers as competing explanations; verify powers by historical period	None
Gemini	Bullets	Critical	Scope overload; mixed eras; decentralisation theory	None
Kooko	Numbered list	Firm but balanced	1999 baseline belongs to Scotland and Wales; unsourced claims	None
Oreate	Prose	Constructive	Falsifiable question; unit of analysis; named confounders	None
Paperguide	Prose	Constructive	Literature protocol as note-taking; three-step fix	None
Atlasworkspace	Table	Critical	Statutes; concrete narrowing; missing combined authorities	Statutory knowledge
Consensus	Short paragraph	Brief	Link to published literature	Literature citations
Claude	Long sections	Firm but balanced	1999 baseline; Local Government Association category error; missing actors	None

4. Were the analyses identical?

No. They were similar in their conclusions and different in almost everything else.

Same verdict, different tone. All ten concluded that the plan needs fundamental development, but the register varied widely. Chaos and SCIspace were severe (“no examiner will let the paper past the proposal stage”; it does not meet minimum standards in any relevant discipline). ChatGPT, Oreate and Paperguide were constructive and led with strengths. Kooko and Claude sat in between, offering praise and then a firm verdict that the plan would not pass proposal review as it stands.

Different format. The same task produced flowing prose (ChatGPT, SCIspace, Chaos), bullets (Gemini), a numbered list (Kooko), a table (Atlasworkspace), a short cited paragraph (Consensus) and a long sectioned report (Claude, though that entry is the extended version produced after a follow-up request, so its length is not comparable to the others).

Genuine disagreement. The most interesting divergence concerns the governance and funding section. ChatGPT and Oreate called it the strongest part of the plan, specific and promising. Kooko said it states contested claims without sources. Claude said it contains category errors. Atlasworkspace said it is outdated. These are not contradictory in every case, since a section can be specific and still be inaccurate, but they show that the same material can be read as the plan’s backbone or its weak point depending on the agent’s priorities.

Different levels of severity on the drivers. Some agents said the list presupposes its conclusions (Chaos, Kooko, Atlasworkspace, Claude). Others said the list is a useful starting point if reframed as hypotheses (ChatGPT, Oreate).

5. Common patterns in AI work behaviour

Reading the ten summaries together reveals habits that look like the working style of AI reviewers in general, not just of individual tools.

Role compliance. Every agent adopted the professor stance and gave a verdict, usually phrased in examiner language (“would be rejected at proposal stage”).
The strengths-then-weaknesses frame. Most opened with a courtesy about the topic’s relevance before turning to flaws.
Diagnosis followed by an ordered remedy. Nearly all ended with numbered or prioritised steps. The typical order was question, definitions, theory, method, data, and literature.
Convergence on textbook research-design categories. The agents mostly assessed the plan against a standard checklist: question, framework, method, causality, measurement, scope. That is reliable and useful, but it means the shared findings are partly a reflection of the checklist and not independent discoveries.
Neutrality policing. Many noticed the pejorative wording of the drivers (corruption, failure, lack of knowledge) and asked for neutral, testable phrasing.
Compression into a single overall judgement. The shorter the summary, the more it relied on a single memorable verdict.

6. Signals of depth in the summaries

Can depth be judged from a summary alone? Only partly, since a summary compresses the full analysis, but several signals are visible.

Stronger signals:

Domain-specific correction. Pointing out that 1999 is the Scottish and Welsh settlement, not an English one, requires knowing the subject and not merely the format of a proposal. Kooko and Claude did this, and Gemini and Paperguide approached it through the mixing of eras.
Named institutions and statutes. Atlasworkspace’s reference to specific Acts, and Claude’s mention of the Greater London Authority, Homes England and the 2004 North East referendum, show retrieval of subject knowledge.
Named theory. Multi-level governance, state rescaling, historical institutionalism and decentralisation theory appear in several entries. The signal is stronger when the theory is tied to a part of the plan (for example, the drivers list) and not just listed.
Naming the confounders. Oreate and Claude named specific disruptors (the financial crisis, austerity, Brexit, COVID-19). Saying “causal logic is weak” is generic. Saying why is specific.
Falsifiability and counterfactuals. Asking what would count as evidence, and against what comparator, reflects a methodological understanding of research.
Concrete narrowing. Proposing a bounded case (post-2016 combined authorities, strategic housing) shows the agent engaged with feasibility, not just with criticism.
External grounding. Consensus’s literature references show verification against published work, even though its summary was the briefest.

Weaker signals:

Statements that could be applied to any proposal in any field (“requires further development”, “needs a sharper question”).
Verdicts without examples from the plan.
Hedging that defers the substantive work to the researcher (“must be verified”) without saying what to verify.

On these signals, the stronger summaries combined a verdict, plan-specific evidence, subject knowledge and an ordered fix. Length alone was not a signal. Paperguide’s short summary carried a precise diagnosis and a three-step fix, while a longer text can be mostly restatement.

A currency check. English devolution law is still moving. The most recent statute in any of the ten summaries is the 2023 Act (Atlasworkspace). Since then the English Devolution and Community Empowerment Act 2026 has received Royal Assent (29 April 2026), restructuring the role of combined authorities through a standardised devolution framework. None of the ten summaries mentions it. This is not a criticism of any single agent, because summaries are short and the plan itself stops earlier, but it shows why a reviewer who checks the currency of the law is worth more than one who only reasons well. The Act is documented on the UK Parliament bill page, and its policy background is set out in the government’s English Devolution White Paper. For the practical limits of current powers, including the fiscal gap, see the House of Commons committee’s oral evidence on English devolution.

7. Limits of this comparison

This is an exploratory case study, and its limits matter:

It examines summaries, not the full responses, so some depth is hidden.
Each agent was run once. AI output varies between runs, so a second run might emphasise different points.
The versions of the input file may have differed (collapsed or expanded branches), which would change what each agent could see.
The tools are different kinds of products, including general chat assistants and research-oriented tools, so a direct league table would be unfair.
One plan, one topic. Results may not transfer to other disciplines.
No blind scoring was applied. Any ranking of depth is an informed reading, not a measurement.

8. Practical guidance for researchers using AI

Check what the AI actually saw. Export the file so nothing is hidden. A mind map with collapsed branches, as in my case, can quietly remove the very details you want criticised.
Assign a role and a standard. Every agent responded to the professor role. Add the standard (proposal review, thesis examination, journal submission) to sharpen the verdict.
Ask for plan-specific evidence. Request that each criticism cite the exact part of the plan. This quickly separates real reading from generic feedback.
Use more than one agent. Convergence across tools is reassuring, and divergence (such as the governance section here) points to the places where your own judgement is most needed.
Match the tool to the job. A retrieval-based tool is better for checking claims against literature, and a general assistant is better for restructuring an argument.
Verify institutional and legal details yourself. Statutes, dates and the powers of bodies change across periods. Treat names and dates from any AI as leads to confirm in primary sources.
Ask for a ranked action list. The most useful summaries ended in prioritised steps.
Expect polite framing. Strengths are often included as a courtesy. Look for the verdict and the ordered fixes.
Keep the final judgement human. AI critique is strongest at structure and design gaps, and weakest where it requires local knowledge, field politics or your own research aims.

Conclusion

Ten different agents, given the same plan and the same instruction, reached the same central conclusion: a promising topic without a research design. That agreement is encouraging, because it suggests that AI reviewers reliably detect the structural gaps that examiners look for. The differences are where the value lies. Some agents corrected the history, some demanded falsifiable questions, one brought in literature, and one reorganised the critique into a table of remedies. For anyone choosing an AI to support academic work, the practical question is not which tool is best in general but what each tool is best at: spotting methodological holes, correcting subject knowledge, grounding claims in literature, or turning criticism into a plan of action. Using several of them on the same draft, and checking what each actually read, produces a far more rigorous review than relying on any single one.

Appendix A: The original task given to the AI agents

I am going to upload a research plan for writing a paper on a subject of urban planning, I want you to take the role of an academic professor and criticize this plan in terms of academic rigor and provide a summary for your critic – the uploaded file is a web base file that include the research paper plan as a mind map

Appendix B: The research plan file

the research plan file uploaded to AI agents as mind map

External links

FAQ

Q: Can AI review an academic research plan?

Yes. In this comparison all ten AI agents correctly identified the plan’s central weakness: it was a topic map without a research question, method or theoretical framework. They are strong at structural and design gaps. They are weaker where local knowledge or very recent law is needed.

Q: Which AI is best for academic research?

There is no single best tool. Retrieval-based tools are better at tying claims to published literature, and general assistants are better at restructuring an argument or correcting concepts. Using more than one on the same draft gives the most reliable review.

Q: Do different AI agents give the same critique?

They reached the same verdict but differed in tone, format, depth and emphasis. They disagreed, for example, on whether the plan’s governance and funding section was its strongest part or a weak point.

Q: How do I get a more useful critique from an AI?

Give it a role and a standard (such as proposal review), make sure the file is fully visible with no collapsed content, and ask it to cite the exact part of the plan for every criticism and to end with a ranked action list.

Q: Can I trust the laws, dates and references an AI gives me?

Treat them as leads, not facts. Statutes and institutional powers change over time, so confirm them in primary sources such as the UK Parliament and GOV.UK websites.