A Comprehensive Overview of Large Language Models

This repo is for our paper: https://arxiv.org/abs/2307.06435

Please cite the paper, if our work is useful to your research:

@article{naveed2023comprehensive,
  title={A Comprehensive Overview of Large Language Models},
  author={Naveed, Humza and Khan, Asad Ullah and Qiu, Shi and Saqib, Muhammad and Anwar, Saeed and Usman, Muhammad and Barnes, Nick and Mian, Ajmal},
  journal={arXiv preprint arXiv:2307.06435},
  year={2023}
}

Contents

• Surveys
• Pre-trained LLMs
  • General Purpose
  • Coding
  • Scientific Knowledge
  • Dialog
  • Finance
• Fine-tuned LLMs
  • Instruction-tuning with Manually Created Datasets
  • Instruction-tuning with LLMs Generated Datasets
  • Aligning with Human Preferences
    • Aligning with Supported Evidence
    • Aligning Directly with SFT
    • Aligning with Synthetic Feedback
    • Aligning with Prompts
    • Red-Teaming Jailbreaking Adversarial Attacks
  • Continue Pre-Training
  • Sample Efficiency
• Increasing Context Window
  • Position Interpolation
  • Efficient Attention Mechanism
  • Extrapolation without Training
• Augmented LLMs
  • Retrieval Augmented LLMs
  • Tool Augmented LLMs

Surveys

• Towards Reasoning in Large Language Models: A Survey, arXiv, 2022. [Paper]
• Emergent Abilities of Large Language Models, arXiv, 2022. [Paper]
• Several categories of Large Language Models (LLMs): A Short Survey arXiv, 2023. [Paper]
• Retrieving Multimodal Information for Augmented Generation: A Survey, arXiv, 2023. [Paper]
• Large Language Models in Medical Education: Opportunities, Challenges, and Future Directions, JMIR, 2023. [Paper]
• Language Model Behavior: A Comprehensive Survey, arXiv, 2023. [Paper]
• Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond, arXiv, 2023. [Paper]
• Beyond One-Model-Fits-All: A Survey of Domain Specialization for Large Language Models, arXiv, 2023. [Paper]
• A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical Usage, TechRxiv, 2023. [Paper]
• Recent advances in natural language processing via large pre-trained language models: A survey, ACM Surveys, 2021. [Paper]
• Complex QA and language models hybrid architectures, Survey, arXiv, 2023. [Paper]
• Challenges and Applications of Large Language Models, arXiv, 2023. [Paper]
• Augmented Language Models: a Survey, arXiv, 2023. [Paper]
• A Survey on Multimodal Large Language Models, arXiv, 2023. [Paper]
• A Survey on Evaluation of Large Language Models, arXiv, 2023. [Paper]
• A Survey of Large Language Models, arXiv, 2023. [Paper]
• ChatGPT for good? On opportunities and challenges of large language models for education, LID, 2023. [Paper]
• A Short Survey of Viewing Large Language Models in Legal Aspect, arXiv, 2023. [Paper]
• Aligning Large Language Models with Human: A Survey, arXiv, 2023. [Paper]
• A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT, arXiv, 2023. [Paper]
• Instruction Tuning for Large Language Models: A Survey, aeXiv, 2023. [Paper]
• Examining User-Friendly and Open-Sourced Large GPT Models: A Survey on Language, Multimodal, and Scientific GPT Models, arXiv, 2023. [Paper]
• Foundation Models for Decision Making: Problems, Methods, and Opportunities, arXiv, 2023. [Paper]
• How Can Recommender Systems Benefit from Large Language Models: A Survey, arXiv, 2023. [Paper]
• A Survey on Large Language Model based Autonomous Agents, arXiv, 2023. [Paper]
• The Rise and Potential of Large Language Model Based Agents: A Survey, arXiv, 2023. [Paper]
• A Survey on Large Language Model based Autonomous Agents, arXiv, 2023. [Paper]
• Beyond One-Model-Fits-All: A Survey of Domain Specialization for Large Language Models, arXiv, 2023. [Paper]
• Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys. [Paper]

Pre-trained LLMs

General Purpose

• T5: Exploring the limits of transfer learning with a unified text-to-text transformer, JMLR, 2020. [Paper]
• GPT-3: Language Models are Few-Shot Learners, NeurIPS, 2020. [Paper]
• mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer, NAACL, 2021. [Paper]
• PanGu-alpha: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation, arXiv, 2021. [Paper]
• CPM-2: Large-scale cost-effective pre-trained language models, AI Open, 2021. [Paper]
• Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. arXiv, 2021. [Paper]
• JURASSIC-1: Technical Details and Evaluation, White Paper, 2021.
• HyperCLOVA: What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers, arXiv, 2021. [Paper]
• Yuan 1.0: Large-scale pre-trained language model in zero-shot and few-shot learning, arXiv, 2021. [Paper]
• Gopher: Scaling language models: Methods, analysis & insights from training gopher, arXiv, 2021. [Paper]
• Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation, arXiv, 2021. [Paper]
• Gpt-neox-20b: An open-source autoregressive language model, arXiv, 2022. [Paper]
• Opt: Open pre-trained transformer language models, arXiv, 2022. [Paper]
• Bloom: A 176b-parameter open-access multilingual language model, arXiv, 2022. [Paper]
• Glam: Efficient scaling of language models with mixture-of-experts, ICML, 2022. [Paper]
• MT-NLG: Using deepspeed and megatron to train megatron-turing nlg 530b, a large-scale generative language model, arXiv, 2022. [Paper]
• Chinchilla: Training compute-optimal large language models, arXiv, 2022. [Paper]
• Alexatm 20b: Few-shot learning using a large-scale multilingual seq2seq model, arXiv, 2022. [Paper]
• Palm: Scaling language modeling with pathways, arXiv, 2022. [Paper]
• U-Palm: Transcending scaling laws with 0.1% extra compute, arXiv, 2022. [Paper]
• Ul2: Unifying language learning paradigms, ICLR, 2022. [Paper]
• Glm-130b: An open bilingual pre-trained model, arXiv, 2022. [Paper]
• Llama: Open and efficient foundation language models, arXiv, 2023. [Paper]
• PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computing, arXiv, 2023. [Paper]

Coding

• Codegen: An open large language model for code with multi-turn program synthesis, arXiv, 2022. [Paper]
• Codex: Evaluating large language models trained on code, arXiv, 2021. [Paper]
• Alpha Code: Competition-level code generation with alphacode, Science, 2022. [Paper]
• Codet5+: Open code large language models for code understanding and generation, arXiv, 2023. [Paper]
• StarCoder: may the source be with you!, arXiv, 2023. [Paper]

Scientific Knowledge

• Galactica: A large language model for science, arXiv, 2022, [Paper]

Dialog

• Lamda: Language models for dialog applications, arXiv, 2022. [Paper]

Finance

• Bloomberggpt: A large language model for finance, arXiv, 2023. [Paper]
• XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters, arXiv, 2023. [Paper]

Fine-tuned LLMs

Instruction-tuning with Manually Created Datasets

• T0: Multitask prompted training enables zero-shot task generalization, arXiv, 2021. [Paper]
• mT0: Crosslingual generalization through multitask fine-tuning, arXiv, 2022. [Paper]
• Tk-Instruct: Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks, arXiv, 2022. [Paper]
• Opt-iml: Scaling language model instruction meta learning through the lens of generalization, arXiv, 2022. [Paper]
• Flan: Scaling instruction-finetuned language models, arXiv, 2022. [Paper]
• The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning, arXiv, 2023. [Paper]
• From zero to hero: Examining the power of symbolic tasks in instruction tuning, arXiv, 2023. [Paper]

Instruction-tuning with LLMs Generated Datasets

• Self-instruct: Aligning language model with self generated instructions, arXiv, 2022. [Paper]
• Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation, arXiv, 2023. [Paper]
• Stanford Alpaca: An Instruction-following LLaMA model, Github, 2023. [Link]
• Vicucna: Github, 2023. [Link]
• LLaMA-GPT-4: INSTRUCTION TUNING WITH GPT-4, arXiv, 2023. [Paper]
• Goat: Fine-tuned LLaMA Outperforms GPT-4 on Arithmetic Tasks, arXiv, 2023. [Paper]
• Huatuo: Tuning llama model with chinese medical knowledge, arXiv, 2023. [Paper]
• Wizardlm: Empowering large language models to follow complex instructions, arXiv, 2023. [Paper]
• WizardCoder: Empowering Code Large Language Models with Evol-Instruct, arXiv, 2023. [Paper]

Aligning with Human Preferences

• InstructGPT: Training language models to follow instructions with human feedback, NeurIPS, 2022. [Paper]
• LLaMA-2-Chat: Llama 2: Open foundation and fine-tuned chat models, arXiv, 2023. [Paper]

Aligning with Supported Evidence

• Webgpt: Browser-assisted question-answering with human feedback, arXiv, 2021. [Paper]
• Sparrow: Improving alignment of dialogue agents via targeted human judgments, arXiv, 2022. [Paper]
• GopherCite: Teaching language models to support answers with verified quotes, arXiv, 2022. [Paper]

Aligning Directly with SFT

• DPO: Direct preference optimization: Your language model is secretly a reward model, arXiv, 2023. [Paper]
• Raft: Reward ranked finetuning for generative foundation model alignment, arXiv, 2023. [Paper]
• Rrhf: Rank responses to align language models with human feedback without tears, arXiv, 2023. [Paper]
• PRO: Preference ranking optimization for human alignment, arXiv, 2023. [Paper]
• CoH: Languages are rewards: Hindsight finetuning using human feedback, arXiv, 2023. [Paper]

Aligning with Synthetic Feedback

• Constitutional ai: Harmlessness from ai feedback, arXiv, 2022. [Paper]
• Alpacafarm: A simulation framework for methods that learn from human feedback, arXiv, 2023. [Paper]
• Self-align: Principle-driven self-alignment of language models from scratch with minimal human supervision, arXiv, 2023. [Paper]

Aligning with Prompts

• Prompting gpt-3 to be reliable, arXiv, 2022. [Paper]
• The capacity for moral self-correction in large language models, arXiv, 2023. [Paper]

Red-Teaming Jailbreaking Adversarial Attacks

• Red teaming language models with language models, arXiv, 2023. [Paper]
• Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned, arXiv, 2022. [Paper]
• Jailbroken: How does llm safety training fail?, arXiv, 2023. [Paper]
• Explore, Establish, Exploit: Red Teaming Language Models from Scratch, arXiv, 2023. [Paper]

Continue Pre-Training

• Fine-tuned language models are continual learners, EMNLP, 2023. [Paper]
• Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner, arXiv, 2023. [Paper]

Sample Efficiency

• Instruction Tuned Models are Quick Learners, arXiv, 2023. [Paper]
• Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning, arXiv, 2023. [Paper]
• Lima: Less is more for alignment, arXiv, 2023. [Paper]

Increasing Context Window

Position Interpolation

• Extending context window of large language models via positional interpolation, arXiv, 2023. [Paper]
• Giraffe: Adventures in Expanding Context Lengths in LLMs, arXiv, 2023. [Paper]
• YaRN: Efficient Context Window Extension of Large Language Models, arXiv, 2023. [Paper]

Efficient Attention Mechanism

• LongT5: Efficient text-to-text transformer for long sequences, NAACl, 2022. [Paper]
• Colt5: Faster long-range transformers with conditional computation, arXiv, 2023. [Paper]
• Longnet: Scaling transformers to 1,000,000,000 tokens, arXiv, 2023. [Paper]
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models, arXiv, 2023. [Paper]

Extrapolation without Training

• LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models, arXiv, 2023. [Paper]
• PCW: Parallel context windows for large language models, ACL, 2023. [Paper]

Augmented LLMs

Retrieval Augmented LLMs

• Retrieval augmented language model pre-training, ICML,2020. [Paper]
• Rationale-augmented ensembles in language models, arXiv, 2022. [Paper]
• RETRO: Improving language models by retrieving from trillions of tokens, ICML, 2022. [Paper]
• Learning to retrieve prompts for in-context learning, NACCL, 2022. [Paper]
• Internet-augmented dialogue generation, ACL, 2022. [Paper]
• Long time no see! open-domain conversation with long-term persona memory, arXiv, 2022. [Paper]
• Internet-augmented language models through few-shot prompting for open-domain question answering, arXiv, 2022. [Paper]
• FLARE: Active retrieval augmented generation, arXiv, 2023. [Paper]
• In-context retrieval-augmented language models, arXiv, 2023. [Paper]
• Repocoder: Repository-level code completion through iterative retrieval and generation, arXiv, 2023. [Paper]
• Shall we pretrain autoregressive language models with retrieval? a comprehensive study, arXiv, 2023. [Paper]
• Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2023. [Paper]
• What makes good in-context examples for GPT-3?, arXiv, 2023. [Paper]
• Learning to Retrieve In-Context Examples for Large Language Models, arXiv, 2023. [Paper]
• Replug: Retrieval-augmented black-box language models, arXiv, 2023. [Paper]
• RPT: Long-range Language Modeling with Self-retrieval, arXiv, 2023. [Paper]
• Fid-light: Efficient and effective retrieval-augmented text generation, SIGIR, 2022. [Paper]
• Augmenting Language Models with Long-Term Memory, arXiv, 2023. [Paper]
• MemoryBank: Enhancing Large Language Models with Long-Term Memory, arXiv, 2023. [Paper]
• Reflexion: Language Agents with Verbal Reinforcement Learning, arXiv, 2023. [Paper]
• ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory, arXiv, 2023. [Paper]
• Memory augmented large language models are computationally universal, arXiv, 2023. [Paper]
• RET-LLM: Towards a General Read-Write Memory for Large Language Models, arXiv, 2023. [Paper]
• Atlas: Few-shot Learning with Retrieval Augmented Language Models, JMLR, 2023. [Paper]

Tool Augmented LLMs

• Talm: Tool augmented language models, arX0v, 2022. [Paper]
• AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn, arXiv, 2023. [Paper]
• Chameleon: Plug-and-play compositional reasoning with large language models, arXiv, 2023. [Paper]
• Art: Automatic multi-step reasoning and tool-use for large language models, arXiv, 2023. [Paper]
• Tool documentation enables zero-shot tool-usage with large language models, arXiv, 2023. [Paper]
• RestGPT: Connecting Large Language Models with Real-World Applications via RESTful APIs, arXiv, 2023. [Paper]
• ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings, arXiv, 2023. [Paper]
• Gorilla: Large language model connected with massive apis, arXiv, 2023. [Paper]
• On the Tool Manipulation Capability of Open-source Large Language Models, arXiv, 2023. [Paper]
• Toolllm: Facilitating large language models to master 16000+ real-world apis, arXiv, 2023. [Paper]
• Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface, arXiv, 2023. [Paper]
• Gpt4tools: Teaching large language model to use tools via self-instruction, arXiv, 2023. [Paper]
• Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis, arXiv, 2023. [Paper]
• Vipergpt: Visual inference via python execution for reasoning, arXiv, 2023. [Paper]