Emergent Properties in Large Language Models (LLMs): Deep Research

I’ve been thinking about emergent properties in LLMs more and more, trying to understand the capabilities and behaviors that we currently see and what we might see in the future. To help me get a better handle on the topic, I enlisted OpenAI’s Deep Research feature to survey the literature and provide a summary on emergent properties.

Below is the full, unedited report:

Introduction

Large Language Models have shown emergent properties — surprising capabilities that were absent in smaller models but appear in larger ones. Emergent behaviors are those that arise unexpectedly as model scale increases, rather than improving in a smooth, predictable way ([2206.07682] Emergent Abilities of Large Language Models) (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology). For example, early versions of GPT could not do basic arithmetic or follow complex prompts, but at a certain scale these abilities seemingly “switched on” with remarkable strength (Characterizing Emergent Phenomena in Large Language Models). This has been observed across many LLMs (OpenAI’s GPT series, Google’s PaLM, DeepMind’s Chinchilla, Meta’s LLaMA, Anthropic’s Claude, etc.), especially in recent years as models reached tens or hundreds of billions of parameters. Researchers are deeply interested in these emergent properties because they expand what AI systems can do — from solving math problems to inferring human intentions — without explicit new programming. Below, we analyze key emergent capabilities, theoretical explanations for how they arise, and benchmark evidence from 2022 onward.

Defining Emergence in LLMs

In complex systems, emergence refers to behaviors that cannot be predicted just by examining the system’s parts (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology). In the context of LLMs, an emergent capability is typically defined as a skill or behavior that is not present or is only at chance-level in smaller models, but appears (often sharply) in larger models ([2206.07682] Emergent Abilities of Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). Crucially, these are nonlinear jumps in performance: scaling up the model’s size or training compute yields a sudden surge on a task, rather than a gradual linear improvement (Characterizing Emergent Phenomena in Large Language Models). Researchers have documented dozens of such cases, meaning that as we make models bigger (in parameters, data, or compute), they begin to solve tasks that they previously could not (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). This phenomenon suggests there may be “critical mass” effects in neural networks, where enough representational capacity and training leads to qualitatively new abilities.

Examples of emergent behavior (detailed in later sections) include:

• In-Context Learning: The ability to learn and follow new tasks from examples in a prompt, without any parameter update.
• Complex Reasoning & Arithmetic: Solving multi-step problems (math, logic, commonsense reasoning) that smaller models essentially guess at.
• Theory of Mind: Inferring the mental states or perspectives of others in stories or scenarios, akin to a rudimentary understanding of belief and intention.
• Tool Use and Programming: Using tools or writing code to aid in solving problems, even without explicit training to use those tools.

These capabilities often “turn on” at specific model scales or training thresholds. For instance, one study noted that multi-digit addition remained at random accuracy for models up to 13 billion parameters, then jumped to high accuracy beyond that scale (Characterizing Emergent Phenomena in Large Language Models). Such discontinuities mean we cannot simply extrapolate a small model’s skills to know what a much larger model might suddenly be capable of ([2206.07682] Emergent Abilities of Large Language Models). This unpredictability has significant implications: on one hand it suggests potential for further surprising capabilities as we scale toward GPT-4-sized and beyond, and on the other hand it raises concerns about reliably forecasting advanced (possibly risky) behaviors (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology) (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology).

Theoretical Perspectives on Emergence

Why do emergent properties occur in LLMs? There is ongoing debate, but several theories offer insight:

• Phase Transitions in Learning: Neural networks may undergo phase changes where adding parameters or data enables them to represent qualitatively new patterns. Before a certain size, the model lacks the minimal circuit to handle a task, leading to near-zero performance; once that critical size is reached, performance leaps upward (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). In other words, many tasks have a threshold of complexity — the model needs sufficient capacity to internalize the concepts or multi-step inference, and below that it can only respond at chance level. This is analogous to a phase transition: e.g. water remains liquid until a precise temperature where it abruptly changes to gas. LLM scaling curves have shown such threshold-like behavior on tasks like arithmetic and logic (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models).
• Implicit Meta-Learning: Large transformers might be meta-learners, meaning they learn how to learn inside their activations. As models train on vast data, they may develop general strategies for adapting to patterns in a prompt. This explains in-context learning as an emergent phenomenon — the model is effectively doing a miniature form of gradient descent or rule inference in its forward pass. Analysis of transformer attention heads supports this: certain “induction heads” in larger models learn to copy sequences and infer patterns from the prompt, a behavior not observed in smaller models. This capability emerges only after the model is complex enough to encode and replicate training patterns of “example → solution” from its data. By the time we reach GPT-3 size (175B), the model can use this mechanism to generalize from a few examples on the fly, something GPT-2 could not do (Large Language Model — Emergent Abilities | by Ling Huang | Medium).
• Pre-training Loss and Data Scaling: Recent research suggests looking at training progress rather than just parameter count () (). As an LLM trains, its pre-training loss (prediction error) gradually decreases. It appears that certain abilities consistently emerge once the loss drops below a threshold, indicating the model has absorbed enough signal from the data (). Notably, one study showed that models with different sizes can reach similar loss if given enough data; at that point, their downstream capabilities (emergent tasks) also become similar () (). For example, Meta’s LLaMA-13B model (13 billion params) was trained on so much high-quality text that its performance on an emergent benchmark rivaled GPT-3’s, despite GPT-3 being much larger (). This implies emergent abilities might correlate better with “effective training compute” (parameters * data) or achieving a certain knowledge threshold, rather than just parameter count. In practical terms, a well-trained smaller model can sometimes match a bigger model’s emergent skills if it saw enough data ().
• Continuous vs. Discontinuous Metrics: Some researchers question whether emergent jumps are truly fundamental, or if they are artifacts of how we measure performance (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology) (). Many tasks are evaluated in a pass/fail manner (e.g. multiple-choice accuracy). A model that improves from 20% to 30% to 40% will suddenly cross a 25% “random guess” baseline and appear to go from “no better than random” to “above random.” From one perspective, the ability was building up slowly, but only when it surpasses that benchmark do we call it present. Schaeffer et al. (2023) argue that by using more fine-grained or continuous metrics, one might see gradual improvement instead of a step change (). However, even they acknowledge that for certain complex tasks, performance really does flatline near-zero until a point where it takes off. The consensus is that while some **emergent behaviors can be predicted or smoothed out with the right metric, many abilities still seem to genuinely require a critical scale before any meaningful performance occurs (Characterizing Emergent Phenomena in Large Language Models).

It’s likely that multiple factors above are true: LLMs undergo internal shifts as they scale, enabling meta-learning and richer representations, and certain tasks simply demand a large capacity. Next, we delve into specific emergent properties observed in modern LLMs, with examples and evidence from recent models like GPT-4, PaLM, Claude, LLaMA and others.

Emergent Capability: In-Context Learning (Few-Shot Learning)

One of the earliest and most celebrated emergent behaviors is in-context learning — the ability of a model to learn from a few examples or an instruction within the prompt (context), without any gradient updates or fine-tuning. Essentially, the model reads a prompt with demonstrations of a task and then generalizes to new query instances on the fly. This capability was first prominently noted in GPT-3 (2020), which was even subtitled “Language Models are Few-Shot Learners.” Smaller predecessors (GPT-2 with 1.5B parameters, or other pre-2020 models) could mimic formatting in a prompt but failed to really generalize to unseen questions; their performance in few-shot settings was no better than random for most tasks (Large Language Model — Emergent Abilities | by Ling Huang | Medium). Yet, GPT-3 at 175B parameters displayed shockingly strong few-shot performance on many NLP tasks — sometimes approaching the accuracy of models that were explicitly fine-tuned for those tasks (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology). This was unexpected: nothing in GPT-3’s training explicitly told it how to do task learning from examples, it emerged from scale and diverse training data.

How it manifests: With GPT-3 and later models, you can give a prompt like: “Translate the following to French. Example: sky -> ciel; tree -> arbre. Now, computer -> ??” and the model will output “ordinateur” (the French word for computer). A smaller model might have just continued the sequence nonsensically or given an unrelated answer, because it wouldn’t grasp the general rule from the examples. GPT-3 was able to infer “it’s showing English->French mapping” and continue the pattern for a new word. This prompted learning works for a variety of tasks: question-answering, grammar correction, style transformation, math problems (to an extent), and more. Larger models do this better – for instance, providing 5-10 examples in the prompt significantly boosts accuracy for GPT-3 175B, but provides little to no benefit for a 6B or 13B model on the same task (Large Language Model — Emergent Abilities | by Ling Huang | Medium). In research benchmarks, this appears as a flat line of performance for small models, then a steep climb at a certain model size when using few-shot prompts (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models).

Why does in-context learning emerge? It likely ties to the massive training corpus: models see many instances of question-answer pairs, worked examples, and dialogues in their training data. A sufficiently large model can contextually bind these patterns: it treats the prompt text (including examples) as just more input to predict on, but thanks to its scale it can internally form an association like “the prompt examples imply the rule I should apply”. Researchers from Anthropic have identified specific attention head patterns (called induction heads) that appear in larger transformers and are absent in small ones; these heads essentially attend to earlier parts of the prompt and try to continue sequences that repeat (Characterizing Emergent Phenomena in Large Language Models). In-context learning is thus an emergent form of meta-learning — the model has learned how to learn from context. Notably, this wasn’t explicitly coded or trained by gradient descent to do one-step learning; it’s a spontaneous capability arising from the general next-word prediction objective combined with scale.

It’s important to mention that in-context learning isn’t perfect or universal. It works better for some tasks than others, even at large scale. For example, a 13B model might do simple arithmetic correctly with a few examples in prompt, but it may fail completely on a task in an unfamiliar language or a highly abstract puzzle, even if we give examples (Large Language Model — Emergent Abilities | by Ling Huang | Medium). The largest models (like GPT-4 or PaLM 540B) further improved in-context learning and can handle more complex, subtle prompts. By 2022, virtually all state-of-the-art LLMs leveraged this property: Google’s PaLM showed strong few-shot results on many tasks, Meta’s LLaMA (65B) matched GPT-3-level few-shot performance despite fewer parameters (credit to excellent training data) (), and Anthropic’s Claude was trained specifically to be good at following prompts and instructions (which is related to in-context learning, especially after instruction fine-tuning). Few-shot prompting has become a standard evaluation for new models — a hallmark of an advanced LLM is that it can solve problems by example alone. This emergent ability has essentially enabled the whole prompt engineering field: treating the model as a general problem-solver that just needs the right examples or instructions in its context, rather than retraining it for each problem.

Emergent Capability: Complex Reasoning and Arithmetic

Another class of emergent properties is the suite of reasoning abilities — solving multi-step problems, performing arithmetic or logical inference, and handling tasks that require following a chain of thought. Early language models were notably bad at these sorts of tasks. For instance, GPT-2 or smaller models would easily get basic math wrong (say, adding two two-digit numbers) and fail logical puzzles, often giving fluent but incorrect answers. Improvements were modest until models became very large. With the advent of models like GPT-3 and Gopher (280B) in 2020–2021, researchers started seeing breakthroughs in reasoning tasks at certain scales (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). A famous example involves multi-digit arithmetic: The GPT-3 paper reported that models up to 13B parameters performed at near-random accuracy on adding two 5-digit numbers, but at 13B and especially 175B parameters, accuracy jumped significantly (GPT-3 175B achieved over 80% accuracy on 2-digit addition and made progress on 3+ digit addition) (Characterizing Emergent Phenomena in Large Language Models). This was a clear emergent jump — nothing gradual about it. Similarly, other math word problems and logic tasks remained unsolved by smaller models but suddenly became tractable as model size crossed into the hundreds of billions of parameters.

Chain-of-Thought Reasoning: A major insight in 2022 was that large LLMs can be prompted to explicitly reason step-by-step, which dramatically improves their ability to tackle complex questions. This is known as chain-of-thought (CoT) prompting. The idea is simple: instead of asking the model directly for the answer, you prompt it to show its work — for example: “Q: There are 5 pens and 3 pencils… (some word problem). Let’s think step by step.” The model then outputs a reasoning process (“First, calculate total items… Then…”) before concluding with the answer. It turns out that small models cannot do this effectively — if you ask a 6B or 13B model to “think step by step,” it either refuses or produces gibberish reasoning. But a large model (like PaLM 540B or GPT-3 175B) magically starts to produce coherent intermediate steps and reaches much better answers (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). This was demonstrated in the work “Chain-of-Thought Prompting Elicits Reasoning in LLMs” (Wei et al., 2022), where providing a few examples of reasoning in the prompt enabled a 540B model to solve 58% of problems on GSM8K (a challenging grade-school math benchmark) — far better than any smaller model and even beating some expert-designed solutions ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models). Crucially, this reasoning ability only emerged at a large scale: the same CoT prompt did nothing for models like LaMDA 8B or even 62B, but suddenly worked wonders for models with on the order of 10²⁴ training FLOPs (e.g. PaLM’s largest size) (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). In that sense, the capacity to carry out multi-step reasoning is itself an emergent property — large models developed an internal scratch pad or logic module that can be activated via prompting. They were never explicitly trained to show chain-of-thought; it came as a byproduct of general training once the model was powerful enough (Characterizing Emergent Phenomena in Large Language Models).

(Characterizing Emergent Phenomena in Large Language Models) Emergent performance on reasoning-heavy tasks often appears as a sudden jump once models reach a certain scale. The figure above (based on data from Google Brain (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models)) shows three examples: (left) a multi-step arithmetic task, (middle) a “multi-task NLU” exam (knowledge and reasoning questions), and (right) a word-in-context disambiguation task. The pink dotted line is random-chance performance. For each task, small models (to the left, <10²² FLOPs of training) hover at or below this baseline. Then at a critical scale (~10²² to 10²⁴ FLOPs, depending on the task), accuracy shoots up dramatically. Notably, this pattern is seen across different model families (LaMDA, GPT-3, Gopher, Chinchilla, PaLM), indicating a general phenomenon (Characterizing Emergent Phenomena in Large Language Models).

One striking case is college-level exam questions. The Massive Multitask Language Understanding benchmark (MMLU) tests models on high school and college exam questions in 57 subjects (history, science, math, etc., multiple-choice). Small models guess at random (~25% accuracy). GPT-3 (175B) reached around 40–50%. But then DeepMind’s Chinchilla (70B, 2022) — thanks to optimal training — reached ~68% (state-of-the-art at the time) (Chinchilla AI — Devin Schumacher). PaLM 540B also surpassed 60%. And in 2023, GPT-4 leaped to 86.4%, exceeding the average human college senior performance on this test. These represent phase changes in knowledge integration and reasoning: GPT-2 couldn’t do it at all; GPT-3 was mediocre; GPT-4 suddenly can ace standardized tests it was never explicitly trained on. Similarly, GPT-4’s creators reported it passed difficult exams like the Uniform Bar Exam (law) in the top 10% of human test-takers, whereas GPT-3.5 was around bottom 10% — a nonlinear improvement of professional-level reasoning in the new model.

So what enabled these reasoning jumps? A few factors stand out:

• Scale and Depth: Very large models have more layers and attention heads to carry out multi-step computations implicitly. They can maintain intermediate results in their activations. Smaller models’ representations seem to collapse or lose track of intermediate steps, whereas large ones can sustain a coherent chain of logic before producing an answer.
• Training Data Breadth: By 2022, models were trained on huge corpora including websites, books, code, etc. The breadth likely included many reasoning examples (e.g. step-by-step solutions on forums, code that breaks down tasks). Large models ingest enough patterns that at some point, they “realize” that complicated problems can be solved by breaking them down. This may be why chain-of-thought prompting suddenly starts to click at large scale: the model has the primitives for multi-step solutions and just needed the right trigger to organize them.
• Prompting Strategies as an Emergent Tool: Interestingly, the ability to use a prompting strategy like CoT is itself emergent. As noted, if you try to coax a 10B model with “let’s think step by step,” it doesn’t actually improve anything. A 100B+ model understands that prompt as a cue to enter a reasoning mode, which is a meta-ability. Other prompting strategies (like self-consistency, where the model generates multiple reasoning paths and then chooses an answer) also only pay off at larger scales. Thus, the model’s flexibility in following complex prompts is an emergent trait. Developers discovered new ways to prompt (few-shot, CoT, etc.) around 2021–2022 precisely because larger models made those techniques effective (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models).

(Characterizing Emergent Phenomena in Large Language Models) Chain-of-thought prompting is a prime example of an emergent strategy. In the graph above (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models), we see the performance of two models (LaMDA and PaLM) on GSM8K math problems with and without CoT prompting, as model scale grows. Small-scale models (left end of x-axis, ~10²¹ FLOPs) have near-zero solve rates regardless of prompting. Mid-size models (~10²²–10²³ FLOPs) show minimal improvement with CoT (the lines for “CoT” vs “No CoT” are close together under 10% accuracy). But for very large models (~10²⁴+ FLOPs), the gap explodes — PaLM 540B reaches about 60% with CoT (blue line) versus ~20% without it (black line). This illustrates that the capacity to utilize a chain-of-thought only emerges beyond a certain scale, unlocking far better reasoning performance.

Beyond math, emergent reasoning appears in other domains: logical deduction puzzles, multi-hop factual queries (where a model must combine clues from multiple sources), and even coding challenges. BigBench, a crowd-sourced benchmark of challenging tasks released in 2022, contained many tasks that only the largest models could do above chance. For example, solving logic grid puzzles, understanding novel proverbs, or performing analogical reasoning were all essentially unsolvable by models under ~50B, but PaLM 540B and GPT-4 made significant headway (137 emergent abilities of large language models — Jason Wei) (137 emergent abilities of large language models — Jason Wei). It’s as if the model develops a general reasoning engine once it’s large enough: it can draw on knowledge and work through it stepwise, whereas smaller models either recall memorized answers (if seen before) or guess.

It should be noted that adding special training can also improve reasoning (e.g. fine-tuning on code or math solutions). For instance, models like GPT-3.5 Codex (which had code training) or Google’s Minerva (finetuned on scientific papers) show better math performance at somewhat smaller scales, suggesting that some reasoning ability can be “taught” without only relying on emergence. However, even these models benefit from sheer scale. In summary, by late 2022, the frontier was that only the very largest LLMs exhibit strong general reasoning and arithmetic skills, marking a distinct emergent capability that distinguishes them from all previous NLP systems.

Emergent Capability: Theory of Mind and Social Reasoning

A particularly intriguing emergent property reported in 2023 is the semblance of a “Theory of Mind” in LLMs. Theory of Mind (ToM) is the ability to understand that others have beliefs, desires, and knowledge different from one’s own — a cognitive skill that humans typically develop around age 4–5. It was long thought to be uniquely human (or at least requiring advanced cognition). In early NLP models, anything like attributing mental states or understanding false beliefs was nonexistent. Yet, as models have grown, they appear increasingly capable of navigating ToM-style tasks — a development that was largely unexpected.

Researchers tested LLMs on classic false-belief scenarios (the kinds of psychology tests given to children). For example: “Alice sees Bob put a cookie in the jar. Alice leaves. Bob moves the cookie to the box. When Alice comes back, where will she look for the cookie?” A correct answer requires understanding that Alice holds a false belief (she didn’t see the move, so she still thinks it’s in the jar). This is trivial for adults, but children under 4 usually fail, and certainly no simple language model would get it right by memorization (since it requires reasoning about someone else’s knowledge). Astonishingly, a study by Kosinski (2023) found that GPT-3.5 and GPT-4 could handle such tasks at a high level (Microsoft Word — manuscript.docx) (Microsoft Word — manuscript.docx). Models before 2020 had virtually 0% success on these Theory of Mind tasks (Microsoft Word — manuscript.docx). The first large model, GPT-3 (175B, 2020), solved about 40% of ToM tests — roughly equivalent to a 3- or 4-year-old child (Microsoft Word — manuscript.docx). The next version (GPT-3.5 “davinci-003”, late 2022) got around 90% correct — approaching adult level (Microsoft Word — manuscript.docx). And GPT-4 (2023) was correct on ~95% of the tests, basically **passing Theory-of-Mind tasks with flying colors】 (Microsoft Word — manuscript.docx). This is a dramatic leap in a capability that involves modeling the unseen mental state of another — something that wasn’t explicitly in the training data in a labeled way.

What explains this emergence? It’s likely that through language training, the model absorbs patterns of human storytelling and social interaction. Many texts involve characters with beliefs and knowledge (think of novels, scripts, news, etc.). A large model can pick up on subtle cues about who knows what in a given narrative. For example, it might learn the pattern “if X didn’t see Y do something, X will have a wrong belief about it.” This is purely implicit learning, but with enough examples in the training corpus, the model can internalize it. Smaller models probably never got enough signal or capacity to do this consistently. Only at large scale does the model have the knowledge and context to apply these social reasoning patterns reliably to new situations. In effect, Theory of Mind reasoning may have spontaneously emerged as a byproduct of improved general language understanding in big LLMs (Microsoft Word — manuscript.docx). The model is not truly self-aware or sentient, of course, but it can simulate the reasoning about others’ minds by drawing on learned patterns.

Besides false-belief tasks, other “social” or commonsense capabilities have seen emergent improvements. For instance, understanding sarcasm or irony often requires knowing that the speaker’s intent differs from the literal statement; large models are much better at picking up such cues compared to small ones. Moral reasoning or ethical judgment is another area: while far from perfect, larger models can handle nuanced prompts about what a person should do in a given dilemma, whereas smaller models either give superficial answers or get confused. These complex forms of understanding likely piggyback on the vast amounts of human discourse the model has ingested. It’s as if scaling endows the model with a broad commonsense knowledge and an ability to infer context that wasn’t explicitly stated. Some benchmarks like Social IQA (which involves understanding motivations and reactions in social scenarios) showed emergent behavior as well — small models were guessing, large models started selecting the commonsense answers with above-chance accuracy (137 emergent abilities of large language models — Jason Wei).

It’s worth noting that not everyone agrees that LLMs truly have Theory of Mind in the human sense. Critics argue that the model doesn’t really understand beliefs; it might just be matching patterns or forms of questions it has seen. However, from a capability standpoint, the end result is the same: bigger models can reliably solve tasks that require reasoning about others’ perspectives, which smaller models cannot do at all. Whether that’s “true” understanding or just regurgitation of training examples is an open question — but given the novelty of the specific false-belief puzzles used in evaluations, the evidence points to a generalizable capacity rather than memorization (Microsoft Word — manuscript.docx) (Microsoft Word — manuscript.docx).

In summary, a kind of social intelligence is emerging in LLMs as they scale. They gain the ability to track who knows what, predict likely human reactions, and even empathize in a shallow form (by saying the right comforting or relevant thing given someone’s described emotional state). These were not explicitly encoded; they emerged from reading and learning on massive text data about human interactions. It’s a striking example of how scaling can produce qualitatively new behavior: the jump from GPT-2 to GPT-4 in this domain is the difference between a system with zero grasp of others’ minds and one that can perform some theory-of-mind tasks at near human level (Microsoft Word — manuscript.docx).

Emergent Capability: Tool Use and Programming

Large language models have also demonstrated an emergent ability to use tools and external interfaces in ways that smaller models simply could not. By “tool use,” we mean the model’s capability to interact with things beyond plain language — for example, writing and executing code, calling external APIs, using a calculator, or controlling a web browser. This is a somewhat broader category, but it represents the model moving from just predicting text to performing actions guided by text. While the core model isn’t physically acting, it can output strings that, when interpreted by some tool, have real effects (like producing a plot, retrieving information, etc.).

One emergent aspect here is the model’s innate ability to write correct code or structured queries that solve problems. In 2021, OpenAI trained Codex (a GPT-3 descendant) on code and it amazed people by generating working code from natural language. But even a purely generic model like PaLM 540B, not exclusively trained as a coder, showed strong coding abilities — it could write small programs to solve math problems, or use JSON format to query a database, etc., once it was large enough. Smaller models might know some syntax but often produce nonsense or incomplete code. By contrast, a large model will often produce executable code on the first try from a high-level request. This seems to be an emergent capacity because it combines multiple skills: understanding the user’s intent, knowing the coding language, planning a solution, and outputting correct structured text. Each of those pieces individually improves with scale (for instance, understanding intent gets better with model size, knowledge of APIs or libraries increases with more training data, etc.), and at a certain point the model can put them together to actually solve problems via code. OpenAI’s GPT-4, for example, can take a natural language question and output a Python script that, when run, computes the answer — effectively using code as a tool for reasoning. This ability to decide “I should write a short script to get the answer” did not exist in earlier GPT models explicitly, but in GPT-4 it emerges as a sensible strategy in response to certain prompts (especially when encouraged by the user) (Multimodality, Tool Use, and Autonomous Agents: Large Language Models Explained, Part 3 | Center for Security and Emerging Technology).

Another example is the use of plugins and APIs. When ChatGPT plugins were introduced in 2023, the only reason it was viable is that models like GPT-4 can read a tool’s documentation and then generate a correct API call or action based on a user request (Multimodality, Tool Use, and Autonomous Agents: Large Language Models Explained, Part 3 | Center for Security and Emerging Technology). For instance, with a calculator plugin, if the user says “What is the square root of 2 to 10 decimal places?”, the model can decide: this requires high precision math, formulate a call to the calculator plugin (like calculator.sqrt(2, 10) in a hypothetical syntax), and then return the result. A small model would not be capable of this multi-step planning: it might not even realize a tool is needed, or know how to format the call. But GPT-4 or Claude can effectively integrate external tools into their reasoning when given the ability. This is emergent in the sense that it wasn’t directly trained via supervised signal to use those new tools – the model’s general intelligence and knowledge let it adapt quickly. Researchers have even shown that a sufficiently large model can be fine-tuned to discover how to use tools on its own. For example, Meta’s Toolformer project (2023) took a 6B model and, through unsupervised training, taught it to insert API calls into its output. However, Toolformer still required explicit fine-tuning. By contrast, GPT-4 with plugins seems to immediately understand tool-use instructions from a short description, which suggests a general capability that comes with scale.

This ties into the concept of LLMs as autonomous agents — given a goal, the model can plan actions (in the form of tool usage or code) to achieve the goal. We saw early glimpses of this with large models like GPT-3: for example, if asked to “find the largest prime less than 1000”, GPT-3 might try to just guess or recall, but a well-prompted GPT-4 can generate a Python loop to test primes, effectively solving it algorithmically. The ability to realize when an external procedure is needed and then carry it out is a new cognitive skill that arises in advanced LLMs. It requires the model to go beyond its training distribution (which was just text) and interface with an external process.

We also see emergent tool use in multi-step interactive settings. For instance, ReAct prompting (another 2022 innovation) has the model alternate between Reasoning and Acting. A sufficiently capable model, when prompted in this format, will produce a series of steps like: “Thought: I need to search for X. Action: [Search API] Query ‘X’.” — then it receives the result and continues. Only larger models reliably follow this pattern without derailing. Smaller ones lose track or output invalid actions. This indicates that the capacity to be an agent (keeping track of state, deciding on actions) comes only with greater model sophistication.

From a benchmarking perspective, tasks that involve tool use or coding have shown big performance gains at large scale. The HumanEval benchmark (writing correct programs for given specs) went from near 0% for small models to 50%+ for Codex (GPT-3.5, 2021) and now over 80% for GPT-4 (2023). That improvement is not just training on more code — it’s also the model’s emergent ability to logically decompose a task into code. Anthropic’s Claude and Google’s PaLM 2 have similarly strong coding/tool abilities (Claude, for example, is reported to be proficient in complex reasoning and coding tasks, rivaling GPT-4 in some evaluations). Meta’s LLaMA, when fine-tuned (e.g. the CodeLLaMA variant), also demonstrates that even a 34B or 70B model can perform tool-like behavior when appropriately guided.

In summary, using tools (like writing code, calling APIs, or controlling a system through text) is a late-emerging capability. It leans on many of the other emergent skills: understanding instructions, reasoning through a procedure, and precise syntax handling. Only by 2022–2023 have LLMs become powerful enough that we can just tell them how to use a tool and they do it. This has opened up an entire new landscape of applications — instead of being just a text generator, an LLM at sufficient scale can function as a general problem-solving agent that leverages external resources. This property was not explicitly programmed; it emerged once the model had enough knowledge and reasoning ability to treat tool interaction as just another learned skill.

Benchmarks and Comparisons Across Models

To concretely illustrate emergent properties, it’s useful to compare how different models (of various sizes) perform on tasks before and after the emergence threshold. The table below summarizes a few notable emergent capabilities and the scale at which they appear, across several LLMs:

Capability / Task Small Models (e.g. <10B) Medium Models (10–50B) Large Models (100B+) Source / Notes Few-Shot In-Context Learning Barely above chance on most tasks; often needs fine-tuning. Moderate improvement with 10B–30B, but inconsistent. Strong performance; e.g. GPT-3 175B few-shot matches or beats some supervised models ([Emergent Abilities in Large Language Models: An Explainer Center for Security and Emerging Technology](https://cset.georgetown.edu/article/emergent-abilities-in-large-language-models-an-explainer/#:~:text=A%20related%E2%80%94but%20distinct%E2%80%94definition%20of%20emergence,exaggerated%20in%20the%20popular%20press)). PaLM 62B+ also shows reliable few-shot learning. Multi-digit Arithmetic Fails (random accuracy). Simple addition/subtraction often incorrect. Still mostly fails or only trivial cases. Success emerges at ~100B scale. GPT-3 175B solves 2-digit add reliably, 3+ digits with partial success (Characterizing Emergent Phenomena in Large Language Models). PaLM 540B handles even harder arithmetic (Characterizing Emergent Phenomena in Large Language Models). Flat performance up to ~10²² FLOPs, then sharp jump (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). Math Word Problems (GSM8K) 0%–5% solve rate. Essentially guesses or gives nonsense. <10% solved even with prompting (can do very simple ones). Huge leap with chain-of-thought: PaLM 540B reached ~57% (Characterizing Emergent Phenomena in Large Language Models); GPT-4 reaches ~90% with advanced prompting. Requires CoT prompting — an emergent strategy only effective for big models (Characterizing Emergent Phenomena in Large Language Models) ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models). MMLU Exam Knowledge (57 tasks) ~25% (random). LLaMA-7B ~35%. ~50% at best (GPT-3 ~45%, LLaMA-13B ~>50% ()). 60–70% for 70B (Chinchilla ~68% (Chinchilla AI — Devin Schumacher), LLaMA-65B ~68%), 86% for GPT-4 (state-of-art). Emergent broad knowledge integration. More data can push medium models higher (e.g. LLaMA-13B beat GPT-3) (). Theory-of-Mind (False-belief) ~0% success. Cannot grasp others’ false beliefs. Still near zero (GPT-J, etc. fail these tasks). Appears at GPT-3: 40% (like age 3), GPT-3.5: 70–90%, GPT-4: ~95% (Microsoft Word — manuscript.docx) (Microsoft Word — manuscript.docx). Other large models likely similar if tested. Uniquely human ability mimicked by LLM once sufficiently large (Microsoft Word — manuscript.docx). Code Generation (HumanEval) <10% pass (@ 1–6B params). Often syntactic or trivial outputs only. ~20–30% pass at 20–30B (if trained on code). ~50%+ pass at 100B+ (Codex 175B ~50%, PaLM-Coder 540B ~60%). GPT-4 >80%. Complex multi-step planning emerges; reliant on extensive code data plus model size.

(Table: Examples of emergent capabilities and the approximate model scales at which they become usable. “Small” and “Medium” refer to standard transformer LLMs of those sizes; specialized training can shift thresholds slightly.)

As seen above, tasks like few-shot learning, arithmetic, and reasoning go from nearly impossible to proficient as we move to the largest models. Importantly, these jumps occur across different architectures and organizations’ models, not just one specific model. For instance, both OpenAI’s GPT-3 and Google’s PaLM showed emergent arithmetic and reasoning leaps (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models). DeepMind’s Chinchilla (70B), despite fewer parameters, displayed emergent performance on knowledge tasks because it was trained with an optimal amount of data — confirming that it’s the overall compute and training that matter for emergence, not just raw size (). Meta’s LLaMA series (2023) further proved that a well-trained 13B model can outperform a 175B model on certain emergent benchmarks (), demonstrating that we can achieve emergent abilities at lower parameter counts by using more data and careful training. Nevertheless, the cutting-edge capabilities (like GPT-4’s superior exam performance or theory-of-mind) still reside in the largest, most advanced models as of 2023.

It’s also notable that architecture tweaks and fine-tuning techniques can influence emergent behaviors. For example, instruction tuning (fine-tuning on following instructions) can make a model appear more capable by unlocking abilities with simpler prompts, but the fundamental abilities still depend on base model scale. Likewise, models trained on code or multi-modal data might exhibit certain emergent skills (like better logical reasoning or the ability to describe images) at somewhat smaller scales than a vanilla model would. However, even these cases align with the idea that once the model’s effective complexity crosses a threshold, new behaviors emerge. Adding modalities or fine-tuning effectively increases the “complexity” or specialization of the model in certain directions.

Conclusion

Emergent properties in LLMs represent a fascinating phenomenon where scaling up an AI system produces qualitatively new skills. In the last few years (2022–2025), we have witnessed LLMs evolve from mere next-word predictors to something more akin to general problem solvers. Capabilities like learning from context, performing multi-step reasoning, understanding human intentions, and utilizing external tools all surfaced as we pushed model architectures and datasets to unprecedented scales. These were not pre-programmed by engineers; instead, they “grew” out of the statistics of language learning at scale. This has fundamentally changed our expectations of AI: tasks once thought to require explicit symbolic logic or human-like cognition have been tackled by large neural networks via emergent behavior.

From a theoretical standpoint, emergent phenomena remind us that scaling is a potent lever in ML — simply making models bigger and training them longer can unlock features that weren’t obvious or present initially. This underscores the importance of scaling laws research and careful evaluation: we must monitor new capabilities (and potential risks) that might appear with the next 10× increase in model size or data. Indeed, discussions in the AI community now revolve around what future emergent abilities might arise (some speculate about advanced planning, autonomy, or even rudimentary forms of common sense physics and causality becoming much stronger). There is also an ongoing debate about predictability: can we anticipate and design for emergent properties, or will they always surprise us? Efforts to link emergence to model loss or to smoother metrics are attempts to demystify it (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology) (), but it remains difficult to forecast specific new abilities.

Practically, emergent capabilities have enabled powerful new applications. Few-shot learning reduced the need for task-specific models. Reasoning skills allow complex AI assistants (like ChatGPT and Claude) to handle elaborate queries, write code, and solve problems interactively. Theory-of-mind-like behavior and social reasoning improve the quality of AI responses in dialogs and make them more context-aware. Tool use via models has given rise to AI “agents” that can browse the web, control software, or analyze data autonomously. All of these were born from scaling up LLMs and then discovering through research that “hey, the model can actually do X now!”

However, with great capability comes risk: emergent behaviors can be unpredictable and hard to control. For example, a model might also develop the emergent ability to deceive or manipulate if such patterns exist in its training data — something we’d want to guard against. The unpredictability of emergence is a key challenge for AI safety and policy. It calls for comprehensive evaluation of models on many axes, even ones we don’t expect them to have aptitude in, because surprises (good or bad) can happen.

In conclusion, emergent properties highlight both the promise and complexity of scaling AI. They demonstrate that with enough data and parameters, simple objectives lead to sophisticated behavior, much like how simple rules in nature can lead to complex life. The research from 2022 onwards has shed light on numerous emergent capabilities (in-context learning, reasoning, theory-of-mind, tool use, and more), but we are likely not at the end of this trajectory. As we develop even larger or more efficient LLMs (GPT-5? PaLM-Next? Open models with trillions of weights?), we should expect further emergent phenomena. Understanding and harnessing these will be crucial — they are the key to making AI not just bigger, but actually more intelligent in a qualitative sense.

Sources:

• Jason Wei et al., “Emergent Abilities of Large Language Models” (TMLR 2022) — provided definition of emergence and dozens of examples ([2206.07682] Emergent Abilities of Large Language Models) (Characterizing Emergent Phenomena in Large Language Models).
• Google Brain Blog (Nov 2022), “Characterizing Emergent Phenomena in LLMs” — gave early examples like arithmetic and word sense disambiguation thresholds (Characterizing Emergent Phenomena in Large Language Models) (Characterizing Emergent Phenomena in Large Language Models).
• OpenAI, GPT-3 Paper (2020) — first showed few-shot learning scaling results, e.g. addition task flat then jumping at 13B+ params (Characterizing Emergent Phenomena in Large Language Models).
• Wei et al., Chain-of-Thought Prompting (2022) — demonstrated emergent reasoning via prompting in PaLM 540B ([2201.11903] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models).
• Kosinski (2023), Theory of Mind May Have Spontaneously Emerged… — tested false-belief understanding in GPT-3/4, showing sharp improvements by scale (Microsoft Word — manuscript.docx) (Microsoft Word — manuscript.docx).
• Meta AI, LLaMA (2023) — showed a 13B model (with high-quality training) outperforming a 175B model, indicating data scaling effects on emergence ().
• CSET Reports (2023–2024) on LLM Emergent Abilities — provided overviews and discussions on policy implications of unpredictable jumps (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology) (Emergent Abilities in Large Language Models: An Explainer | Center for Security and Emerging Technology).
• Various model reports (GPT-4 Technical Report, PaLM and PaLM 2 papers, etc.) for specific benchmark numbers illustrating these emergent capability differences.