A survey on large language model based autonomous agents

This is a systematic review, not an experiment. The authors tested no single intervention. Instead, they:

• Analysed the architectural designs of LLM-based autonomous agents across ~200 papers published between January 2021 and August 2023
• Categorised agent designs into four modules: profiling, memory, planning, and action
• Compared three methods for creating agent profiles: handcrafting, LLM-generation, and dataset alignment
• Reviewed applications in three domains: social science (e.g., simulating human behaviour), natural science (e.g., scientific discovery), and engineering (e.g., software development)
• Evaluated assessment strategies, distinguishing subjective (human judgement) from objective (automated metrics) approaches

The "outcome measures" were qualitative: the presence/absence of specific architectural features, the types of tasks agents could complete, and the evaluation methods used.

No human participants were studied. The "subjects" were:

• ~200 published papers on LLM-based autonomous agents from January 2021 to August 2023
• Specific agent systems reviewed in detail: Generative Agents, MetaGPT, ChatDev, RecAgent, PTLLM, and others
• LLMs used as backbones: primarily GPT-3, GPT-4, ChatGPT, and open-source models like LLaMA

No sample size, demographics, or population characteristics apply — this is a literature review, not a human study.

The authors used a qualitative systematic review methodology with the following approach:

• Literature search: Papers were collected from arXiv, ACL, NeurIPS, ICML, ICLR, and other major AI venues
• Inclusion criteria: Papers proposing or evaluating LLM-based autonomous agents published between January 2021 and August 2023
• Taxonomy development: The authors created a unified framework (profiling → memory → planning → action) and classified each paper according to which modules it used
• Application categorisation: Papers were grouped by domain (social science, natural science, engineering)
• Evaluation strategy analysis: Each paper's evaluation method was classified as subjective (human raters, surveys) or objective (automated metrics, task completion rates)

No quantitative instruments (scales, questionnaires, physiological measures) were used because this is a review of existing literature, not a primary data collection study.

Study design: Systematic review with qualitative synthesis. The authors did not perform a meta-analysis (no quantitative pooling of effect sizes) because the reviewed papers used heterogeneous tasks, metrics, and evaluation criteria.

Search and selection: The authors searched multiple academic databases and preprint servers. They do not report a PRISMA flow diagram, search strings, or explicit inclusion/exclusion criteria beyond the date range (2021–2023) and topic relevance. This is a methodological weakness — without a transparent search strategy, reproducibility is limited.

Data extraction: For each paper, the authors extracted:

• Agent architecture (which modules were used)
• Profile generation method (handcrafted, LLM-generated, or dataset-aligned)
• Application domain
• Evaluation approach (subjective vs. objective)
• Key findings

Synthesis approach: The authors organised findings into a unified framework (Figure 2) and discussed patterns across papers qualitatively. They did not calculate inter-rater reliability, effect sizes, or confidence intervals.

What this design can prove:

• It can identify common architectural patterns across the field
• It can map the landscape of current research and highlight gaps
• It can propose a standardised taxonomy for future work

What this design cannot prove:

• It cannot establish which agent architecture is "best" — no head-to-head comparisons were performed
• It cannot quantify effect sizes or statistical significance of any intervention
• It cannot control for publication bias (papers with positive results are more likely to be published)
• It cannot assess the quality or rigour of individual studies systematically (no risk-of-bias assessment is reported)

Major methodological weaknesses:

• No pre-registered protocol
• No systematic quality assessment of included studies (e.g., no use of ROBINS-I or similar tools)
• No quantitative synthesis or meta-analysis
• The search strategy is not fully reproducible
• The review is descriptive rather than evaluative — it catalogues what exists rather than testing hypotheses

Architecture patterns:

• The unified framework (profiling → memory → planning → action) encompasses "most" previous work, though the authors do not report what percentage of papers fit this framework
• Three profile generation methods were identified: handcrafting (most common), LLM-generation, and dataset alignment
• Memory modules were classified into short-term (within-session) and long-term (cross-session) variants
• Planning modules ranged from simple chain-of-thought prompting to hierarchical task decomposition

Application domains:

• Social science: Agents have been used to simulate human behaviour in social settings (e.g., Generative Agents simulating a small town of 25 agents), study opinion dynamics, and model economic decisions
• Natural science: Agents have been applied to scientific discovery (e.g., ChemCrow for chemistry, BioGPT for biology), though the authors note these are "early-stage"
• Engineering: The most mature application area, with agents used for software development (MetaGPT, ChatDev), code generation, and tool use

Evaluation strategies:

• Subjective evaluation: Human raters assess agent outputs for quality, coherence, or human-likeness. Used in ~40% of reviewed papers (estimated from figures, not explicitly stated)
• Objective evaluation: Automated metrics (e.g., task completion rate, BLEU score, accuracy on benchmarks). Used in ~60% of papers
• No standardised benchmark exists: Different papers use different tasks, making cross-study comparison impossible

Capability acquisition strategies:

• Fine-tuning approaches: Some agents fine-tune LLMs on domain-specific data (e.g., code for programming agents)
• Prompt-based approaches: Most agents use in-context learning (prompt engineering) without modifying model weights
• Tool use: Many agents are equipped with external tools (e.g., web search, calculators, code interpreters) to extend capabilities

Challenges identified:

• Reliability: LLM-based agents can hallucinate, produce inconsistent outputs, or fail on simple tasks
• Safety: Agents acting autonomously could cause harm (e.g., generating malicious code, giving dangerous advice)
• Generalisation: Agents trained/tested in one domain often fail in others
• Evaluation: No consensus on how to measure agent performance

This is a qualitative review, so no effect sizes, confidence intervals, or p-values are reported. The authors do not quantify how much better LLM-based agents perform compared to traditional reinforcement learning agents or rule-based systems.

The closest to a quantitative finding: the cumulative number of papers on LLM-based autonomous agents grew from ~5 in January 2021 to ~200 by August 2023 — a ~40-fold increase in ~2.5 years. This is a bibliometric observation, not an experimental effect.

What the authors acknowledge:

• The field is "rapidly developing" and the review may not capture the most recent work
• The proposed unified framework may not encompass all possible agent architectures
• Evaluation strategies are "not yet mature"
• The review is descriptive rather than prescriptive

What a critical reader would note:

• No systematic quality assessment: The authors do not evaluate the rigour of individual studies. A paper with 5 participants and a paper with 500 are treated equally
• Publication bias: The field is dominated by positive results (agents that work). Failed architectures or negative results are rarely published
• LLM dependence: Most reviewed agents use proprietary LLMs (GPT-3/4). Results may not generalise to open-source models or future model versions
• No replication analysis: The authors do not report whether any findings have been independently replicated
• Industry funding: Many reviewed papers come from tech companies (OpenAI, Google, Meta) with commercial interests in LLMs. The review does not discuss conflicts of interest
• Temporal bias: The review covers only 2021–2023. Given the field's rapid pace, findings may already be outdated
• No negative results: The review focuses on what agents can do, not what they fail at. Failures are mentioned only briefly in the challenges section
• Lack of quantitative synthesis: Without meta-analysis, it's impossible to know which approaches are statistically superior

For someone running their own n=1 experiment with LLM-based autonomous agents:

What to test (specific intervention and dose)

• Intervention: Build an LLM-based autonomous agent using the unified framework (profiling + memory + planning + action modules)
• Dose: Start with a single agent performing one well-defined task (e.g., "write a Python script to scrape website X and save results to CSV"). Do not attempt multi-agent collaboration initially
• Comparison: Compare against (a) doing the task manually, (b) using a simple LLM prompt without agent architecture, or (c) using a traditional rule-based system

Minimum meaningful duration

• Per trial: 1–3 hours for a single task completion
• Total experiment: At least 10–20 trials across different tasks to assess generalisability
• Long-term: If testing memory/learning, run 5–10 sessions over 1–2 weeks, with the agent retaining information across sessions

What to measure (specific metrics)

• Task completion rate: Did the agent complete the task? (binary: yes/no)
• Time to completion: Minutes from start to finish
• Error rate: Number of mistakes (e.g., syntax errors in code, incorrect outputs)
• Number of human interventions: How many times did you need to correct or redirect the agent?
• Output quality: Rate on a 1–5 scale (subjective, but use a rubric: accuracy, completeness, readability)
• Hallucination count: Number of false statements or fabricated information
• Cost: API calls made, tokens used, total cost in USD

Key confounds to control for

• LLM version: Use the same model version throughout (e.g., GPT-4-turbo-2024-04-09). Model updates can change behaviour
• Prompt engineering: Small changes in prompts can cause large changes in output. Document your exact prompts
• Temperature setting: Keep temperature constant (start with 0.2 for deterministic tasks, 0.7 for creative tasks)
• Task difficulty: Vary task difficulty systematically. Don't compare easy tasks to hard tasks
• Order effects: Randomise the order of tasks if comparing multiple conditions
• Learning effects: The agent may "learn" from previous tasks if memory is enabled. Control for this by resetting the agent between conditions
• Human bias: If you're evaluating outputs subjectively, use blinded evaluation (don't know which condition produced which output)

What a positive result would look like

• Task completion rate: ≥80% of tasks completed without human intervention (compared to ≤50% with simple LLM prompt)
• Time savings: Agent completes tasks in ≤25% of the time it takes you manually
• Error reduction: Agent makes ≤1 error per task (compared to ≥3 errors when you do it manually)
• Cost efficiency: Total API cost is less than the value of your time saved (e.g., $2 in API calls saves you 30 minutes of work)
• Consistency: Agent produces similar-quality outputs across 10+ trials (standard deviation in quality ratings ≤0.5 on 1–5 scale)
• Generalisation: Agent succeeds on tasks it wasn't explicitly designed for (e.g., a "code-writing agent" can also debug existing code)

Warning: A single successful trial does not mean the agent is reliable. Run at least 10 trials before drawing conclusions. And remember: LLM-based agents are stochastic — the same prompt can produce different outputs each time. Track this variability.