Personal Second Brain

Most recent

See 344 more →

A Survey of Large Language Models

2 min read

TL;DR

A comprehensive 144-page survey maps the full LLM lifecycle — pre-training, adaptation tuning, utilization, and evaluation — and catalogs open research challenges including hallucination and alignment.

Summary

Beyond a critical parameter scale, language models exhibit emergent capabilities absent in smaller counterparts — qualitatively better reasoning, instruction following, and few-shot generalization. This survey maps the full LLM lifecycle: pre-training (data curation, architecture choices), adaptation tuning (instruction tuning, RLHF, parameter-efficient methods), utilization (in-context learning, chain-of-thought prompting), and capacity evaluation across standard benchmarks. It covers over 1,000 references and closes with open challenges including hallucination, alignment, and inference efficiency.

Key contributions

  1. Provides a structured taxonomy of the full LLM development pipeline from pre-training through deployment as of early 2023.
  2. Establishes that emergent capabilities appear only past a parameter threshold, making scale qualitatively distinct.
  3. Catalogs instruction tuning and RLHF as the primary techniques for converting pre-trained models into usable assistants.
  4. Surveys in-context learning and chain-of-thought prompting as the dominant no-gradient utilization strategies.
  5. Identifies hallucination, alignment, and interpretability as the principal unsolved open challenges.

When to cite

  1. When needing a canonical reference for the LLM development lifecycle or any of its major stages.
  2. When supporting claims about emergent abilities and the qualitative role of model scale.
  3. When discussing instruction tuning or RLHF as alignment techniques for pre-trained models.
  4. When motivating domain-specific evaluation over standard benchmarks for real-world LLM deployment.

LLM GPT Deep Learning Machine Learning Generative AI Prompt Engineering Retrieval-Augmented Generation for Large Language Models - A Survey Prompting Frameworks for Large Language Models - A Survey

Graph View

Backlinks