Jump to content

Language model benchmark

Add links
From Wikipedia, the free encyclopedia

Language model benchmarks are standardized tests designed to evaluate the performance of language models on various natural language processing tasks. These tests are intended for comparing different models' capabilities in areas such as language understanding, generation, and reasoning.

Benchmarks generally consist of a dataset and corresponding evaluation metrics. The dataset provides text samples and annotations, while the metrics measure a model's performance on tasks like question answering, text classification, and machine translation. These benchmarks are developed and maintained by academic institutions, research organizations, and industry players to track progress in the field.

Generally, the life cycle of a benchmark consists of the following steps:[1]

  • Inception: A benchmark is published. It can be simply given as a demonstration of the power of a new model (implicitly) that others then picked up as a benchmark, or as a benchmark that others are encouraged to use (explicitly).
  • Growth: More papers and models use the benchmark, and the performance on the benchmark grows.
  • Maturity, degeneration or deprecation: A benchmark may be saturated, after which researchers move on to other benchmarks.
  • Renewal: A saturated benchmark can be upgraded to make it no longer saturated, allowing further progress.

Construction

[edit]

Like datasets, benchmarks are typically constructed by several methods, individually or in combination:

  • Web scraping: Ready-made question-answer pairs may be scraped online, such as from websites that teach mathematics and programming.
  • Conversion: Items may be constructed programmatically from scraped web content, such as by blanking out named entities from sentences, and asking the model to fill in the blank. This was used for making the CNN/Daily Mail Reading Comprehension Task.
  • Crowd sourcing: Items may be constructed by paying people to write them, such as on Amazon Mechanical Turk. This was used for making the MCTest.

Evaluation methods

[edit]

Generally, benchmarks are fully automated. This limits the questions that can be asked. For example, with mathematical questions, "proving a claim" would be difficult to automatically check, while "calculate an answer with a unique integer answer" would be automatically checkable. With programming tasks, the answer can generally be checked by running unit tests, with an upper limit on runtime.

The benchmark scores are of the following kinds:

  • pass@n: The model is given attempts to solve each problem. If any attempt is correct, the model earns a point. The pass@n score is the model's average score over all problems.
  • cons@n: The model is given attempts to solve each problem. If the most common answer is correct, the model earns a point. The cons@n score is the model's average score over all problems. Here "cons" stands for "consensus" or "majority voting".[2]

The pass@n score can be estimated more accurately by making attempts, and use the unbiased estimator , where is the number of correct attempts.[3]

For less well-formed tasks, where the output can be any sentence, there are the following commonly used scores: BLEU ROUGE, METEOR, NIST, word error rate, LEPOR, CIDEr,[4] SPICE,[5] etc.

Issues

[edit]
  • error: Some benchmark answers may be wrong.[6]
  • ambiguity: Some benchmark questions may be ambiguously worded.
  • subjective: Some benchmark questions may not have an objective answer at all. This problem generally prevents creative writing benchmarks. Similarly, this prevents benchmarking writing proofs in natural language, though benchmarking proofs in a formal language is possible.
  • open-ended: Some benchmark questions may not have a single answer of a fixed size. This problem generally prevents programming benchmarks from using more natural tasks such as "write a program for X", and instead uses tasks such as "write a function that implements specification X".
  • inter-annotator agreement: Some benchmark questions may be not fully objective, such that even people would not agree with 100% on what the answer should be. This is common in natural language processing tasks, such as syntactic annotation.[7][8][9][10]
  • shortcut: Some benchmark questions may be easily solved by an "unintended" shortcut. For example, in the SNLI benchmark, having a negative word like "not" in the second sentence is a strong signal for the "Contradiction" category, regardless of what the sentences actually say.[11]
  • contamination: Some benchmark questions may have answers already present in the training set. Also called "training on the test set".[12][13] Some benchmarks (such as Big-Bench) may use a "canary string", so that documents containing the canary string can be voluntarily removed from the training set.
  • saturation: As time goes on, many models reach the highest performance level practically possible, and so the benchmark can no longer differentiate these models. For example, GLUE had been saturated, necessitating SuperGLUE.
  • Goodhart's law: If new models are designed or selected to score highly on a benchmark, the benchmark may cease to be a good indicator for model quality.[1]
  • cherry picking: New model publications may only point to benchmark scores on which the new model performed well, avoiding benchmark scores that it did badly on.

List of benchmarks

[edit]

Language

[edit]

Question answering

[edit]
  • MCTest (Machine Comprehension Test): 500 fictional stories, each with 4 multiple-choice questions (with at least 2 requiring multi-sentence understanding), designed to be understandable by a 7-year-old. The vocabulary was limited to approximately 8,000 words probably known by a 7-year-old. The stories were written by workers on Amazon Mechanical Turk.[14]
  • SQuAD (Stanford Question Answering Dataset): 100,000+ questions posed by crowd workers on 500+ Wikipedia articles. The task is, given a passage from Wikipedia and a question, find a span of text in the text that answers the question.[15]
  • SQuAD 2.0: 50,000 unanswerable questions written adversarially by crowd workers to look similar to SQuAD questions. Every such unanswerable question must be answered with an empty string.[16]
  • TriviaQA: 650K question-answer-evidence triples. Includes 95K question-answer pairs scraped from 14 trivia and quiz-league websites, and (on average 6) evidence documents for each pair, gathered by searching with Bing and Wikipedia.[17]
  • SearchQA: 140,461 question-answer pairs from the J! Archive, with each pair augmented with (on average 50) snippets and urls obtained by searching the question on Google.[18]
  • ARC (AI2 Reasoning Challenge): Multiple choice questions, with a Challenge Set (2590 questions) and an Easy Set (5197 questions). Designed specifically to supercede SNLI and SQuAD.[19]
  • HotpotQA: 113K multi-hop questions that require reading multiple Wikipedia-based passages to answer. They were produced by showing crowd workers multiple supporting context documents and asking them to produce questions that requiring reasoning about all of the documents.[20]
  • TruthfulQA: 817 questions in health, law, finance and politics with common misconceptions.[21]
  • StrategyQA: 2,780 questions annotated with relevant passages from Wikipedia, such that the question require multi-hop reasoning over the passages to answer. For example, "Did Aristotle use a laptop?" is annotated with passages from the Wikipegia pages for "laptop" and "Aristotle".[22]

Others

[edit]
  • WSC (Winograd schema challenge): 273 sentences with ambiguous pronouns. The task is to determine what the pronoun refers to.[23]
  • WinoGrande: A larger version of WSC with 44,000 items. Designed to be still challenging to the SOTA models of the time (2019) since the original had been saturated. This dataset consists of fill-in-the-blank style sentences, as opposed to the pronoun format of previous datasets.[24][25]
  • SNLI (Stanford Natural Language Inference: 570K human-written English sentence pairs manually labeled for balanced classification with the labels "entailment", "contradiction", and "neutral".[26][27]
  • MultiNLI (Multi-Genre Natural Language Inference): Similarly to SNLI, with 433K English sentence pairs from ten distinct genres of written and spoken English.[28]
  • CNN/Daily Mail Reading Comprehension Task: Articles from CNN (380K training, 3.9K development, 3.2K test) and Daily Mail (879K training, 64.8K development, 53.2K test) were scraped. The bullet point summaries accompanying the news articles were used. One entity in a bullet point was replaced with a placeholder, creating a cloze-style question. The goal is to identify the masked entity from the article.[29]
  • SWAG (Situations With Adversarial Generations): 113K descriptions of activities or events, each with 4 candidate endings; the model must choose the most plausible ending[30]
  • HellaSwag (Harder Endings, Longer contexts, and Low-shot Activities for SWAG): A harder version of SWAG. Contains 10K items.[31][32]
  • RACE (ReAding Comprehension Examinations): 100,000 reading comprehension problems in 28,000 passages, collected from the English exams for middle and high school Chinese students in the age range between 12 to 18.[33]
  • LAMBADA: 10,000 narrative passages from books, each with a missing last word that humans can guess if given the full passage but not from the last sentence alone.[34]

Omnibus

[edit]

Some benchmarks are "omnibus", meaning they are made by combining several previous benchmarks.

  • GLUE (General Language Understanding Evaluation): collection of 9 benchmarks designed for testing general language understanding. The tasks are in the format of sentence- or sentence-pair. There are over 1M items.[35][36]
  • SuperGLUE: An update to GLUE. Designed to be still challenging to the SOTA models of the time (2019) since the original had been saturated. Includes 8 additional tasks (e.g. logical reasoning, commonsense inference, coreference resolution).[37]
  • Big-Bench (Beyond the Imitation Game): A benchmark collection of 204 tasks.[38] A particular subset of 23 tasks is called BBH (Big-Bench Hard).[39]

Agency

[edit]
  • GAIA: 450 questions with unambiguous answers that require information that can be obtained by browsing the Internet, requiring different levels of tooling and autonomy to solve. Divided into 3 difficulty levels.[40]
  • WebArena: 241 mock-up websites based on real-world websites (Reddit, GitLab, Magento's admin portal, etc), and 812 tasks to be performed on the websites. The tasks include information-seeking, site navigation, and content and configuration operation.[41]

Context length

[edit]

Some benchmarks were designed specifically to test for processing continuous text that is very long.

  • Long Range Arena: 6 synthetic tasks that required 1K to 16K tokens of context length to solve.[42]
  • Needle in a haystack tests: Not a specific benchmark, but a method. In this method, a long context window is filled with text, such as Paul Graham's essays, and a random statement is inserted. The task is to answer a question about the inserted statement.[43]
  • L-Eval: 2,000+ human-labeled query-response pairs over 508 long documents in 20 tasks, including diverse task types, domains, and input length (3K--200K tokens).[44]
  • InfiniteBench: 3946 items in 12 tasks from 5 domains (retrieval, code, math, novels, and dialogue) with context lengths exceeding 100K tokens.[45]
  • ZeroSCROLLS: 4,378 items in 6 tasks. Includes 6 tasks from SCROLLS and introduces 4 new datasets. Named "zero" because it was designed for zero-shot learning during the early days of pretraining paradigm, back when zero-shot capability was uncommon.[46]
  • LongBench: 4,750 tasks on 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese).[47] Updated with LongBench v2 that contained 503 more tasks, that require a context length ranging from 8K to 2M words, with the majority under 128K.[48][49]
  • RULER: 13 tasks in 4 categories (retrieval, multi-hop, aggregation, question answering). Each task is specified by a program which can generate arbitrarily long instances of each task on demand.[50]
  • LOFT (Long-Context Frontiers): 6 long-context task categories (text retrieval, visual retrieval, audio retrieval, retrieval-augmented generation, SQL-like dataset query, many-shot in-context learning) in 35 datasets and 4 modalities. Up to 1 million tokens.[51]

Reasoning

[edit]

Mathematics

[edit]
  • Alg514: 514 algebra word problems and associated equation systems gathered from Algebra.com.[52][53]
  • Math23K: 23,164 elementary school Chinese mathematical word problems, collected from various online educational websites.[54]
  • AQuA-RAT (Algebra Question Answering with Rationales): Also known as just "AQuA". 100,000 algebraic word problems with 5 choices per problem, and an annotation for the correct choice with natural language rationales. 34,202 "seed problems" were collected from many sources, such as GMAT and GRE, which were then expanded to the full dataset with Amazon Turk.[55]
  • GSM8K (Grade School Mth): 8.5K linguistically diverse elementary school math word problems that require 2 to 8 basic arithmetic operations to solve.[56]
  • MMLU (Measuring Massive Multitask Language Understanding): 16,000 multiple-choice questions spanning 57 academic subjects including mathematics, philosophy, law, and medicine.[57] Upgraded to MMLU-Pro which increases the number of choices from 4 to 10, eliminated the trivial and noisy questions from MMLU, and added harder problems.[58]
  • MATH: 12,500 competition-level math problems divided into difficulty levels 1 to 5 (as the Art of Problem Solving), with AIME problems being level 5.[59]
  • MathQA: 37,200 word problems in English. Each problem came from AQuA-RAT, and annotated with an "operation program" which exactly specifies the mathematical operations required to solve the problem, written in a domain-specific language with 58 operators.[60] Has a variant, MathQA-Python, consisting of 23,914 problems, produced by taking the solutions to a subset of the MathQA dataset, and rewriting into Python.[61]
  • MathEval: An omnibus benchmark that contains 20 other benchmarks, such as GSM8K, MATH, and the math subsection of MMLU. Over 20,000 math problems. Difficulty ranges from elementary school to high school competition.[62]
  • TheoremQA: 800 questions that test for the use of 350 theorems from math, physics, electric engineering, computer science, and finance.[63]
  • MiniF2F (mini formal-to-formal): 488 Olympiad-level mathematics problems from AIME, AMC, and IMO, stated in formal languages (Metamath, Lean, Isabelle (partially) and HOL Light (partially)).[64]
  • Omni-MATH: 4428 competition-level math problems with human annotation.[65]
  • FrontierMath: Several hundred questions from areas of modern math that are difficult for professional mathematicians to solve. Many questions have integer answers, so that answers can be verified automatically. Held-out to prevent contamination.[66]
  • MathArena: Instead of a purpose-built benchmark, the MathArena benchmark simply takes the latest math competitions (AIME and HMMT) as soon as possible and uses those to benchmark LLMs, to prevent contamination.[67]

Programming

[edit]
  • APPS: 10,000 problems from Codewars, AtCoder, Kattis, and Codeforces.[68]
  • MBPP (Mostly Basic Programming Problems): 974 short Python functions designed to be solved by entry-level programmers. Each comes with a text description and unit tests. They were written by an internal pool of crowdworkers who have basic knowledge of Python.[61]
  • HumanEval: 164 problems where the solution is always a python function, often just a few lines long.[69]
  • CodeElo: 387 contest problems from Codeforces during 2024, annotated with metadata such as contest divisions, problem difficulty ratings, and problem algorithm tags. Benchmarking is run by directly submitting to Codeforces, resulting in an Elo rating. Limited to 8 submissions per problem.[70]
  • SWE-bench: 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase and an issue, the task is to edit the codebase to solve the issue.[71] There are 2 subsets: Lite (300 problems that are faster to run), Verified (human-validated subset of 500 problems reviewed by software engineers).[72]
  • SWE-bench Multimodal: a variant of SWE-bench, with 619 task instances from 17 popular JavaScript repositories, each featuring images that are required for solving the task.[73]
  • SWE-Lancer: 1,488 freelance software engineering tasks from Upwork. Includes implementation tasks (from $50 bug fixes to $32,000 feature implementations) and managerial tasks, where the model must choose between technical implementation proposals.[74][75]
  • KernelBench: 250 PyTorch machine learning tasks, for which a CUDA kernel must be written.[76]

General

[edit]
  • GPQA (Google-Proof Q&A): 448 multiple-choice questions written by domain experts in biology, physics, and chemistry, and requires PhD-level experts to solve. The "Diamond" subset contains the 198 hardest questions in it.[77]
  • SuperGPQA: 26,529 multiple-choice questions collected by domain experts in 285 graduate-level disciplines. The questions were collected by individuals with or pursuing a PhD and then refined and inspected with the help of large language models.[78]
  • AGIEval: questions from 20 official, public, and high-standard admission and qualification exams, such as SAT, Gaokao, law school admission tests, math competitions, lawyer qualification tests, and national civil service exams.[79]
  • OlympicArena: 11,163 problems from 62 distinct Olympic competitions.[80]
  • OlympiadBench: 8,476 math and physics problems in English and Chinese, sourced from International Olympiads, Chinese Olympiads, and Gaokao.[81]
  • ARC-AGI (Abstraction and Reasoning Corpus for Artificial General Intelligence): Given three pairs of before-and-after diagrams of applying a rule, apply the same rule to the fourth before-diagram. It is similar to a Raven's Progressive Matrices test.[82]
  • LiveBench: A series of benchmarks released monthly, including high school math competition questions, competitive coding questions, logic puzzles, and other tasks.[83]
  • Humanity's Last Exam: 3,000 questions across over a hundred academic subjects, with a held-out private dataset left unreleased to prevent contamination. 10% of questions requires both image and text comprehension and the rest are fully text-based. 80% of questions are scored by exact-match, and the rest are multiple-choice.[84]

See also

[edit]
[edit]

References

[edit]
  1. ^ a b Dehghani, Mostafa; Tay, Yi; Gritsenko, Alexey A.; Zhao, Zhe; Houlsby, Neil; Diaz, Fernando; Metzler, Donald; Vinyals, Oriol (2021-07-14), The Benchmark Lottery, arXiv:2107.07002
  2. ^ DeepSeek-AI; Guo, Daya; Yang, Dejian; Zhang, Haowei; Song, Junxiao; Zhang, Ruoyu; Xu, Runxin; Zhu, Qihao; Ma, Shirong (2025-01-22), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948
  3. ^ Chen, Mark; Tworek, Jerry; Jun, Heewoo; Yuan, Qiming; Pinto, Henrique Ponde de Oliveira; Kaplan, Jared; Edwards, Harri; Burda, Yuri; Joseph, Nicholas (2021-07-14), Evaluating Large Language Models Trained on Code, arXiv:2107.03374
  4. ^ Vedantam, Ramakrishna; Lawrence Zitnick, C.; Parikh, Devi (2015). "CIDEr: Consensus-Based Image Description Evaluation": 4566–4575. {{cite journal}}: Cite journal requires |journal= (help)
  5. ^ Anderson, Peter; Fernando, Basura; Johnson, Mark; Gould, Stephen (2016). "SPICE: Semantic Propositional Image Caption Evaluation". In Leibe, Bastian; Matas, Jiri; Sebe, Nicu; Welling, Max (eds.). Computer Vision – ECCV 2016. Lecture Notes in Computer Science. Vol. 9909. Cham: Springer International Publishing. pp. 382–398. doi:10.1007/978-3-319-46454-1_24. ISBN 978-3-319-46454-1.
  6. ^ Northcutt, Curtis G.; Athalye, Anish; Mueller, Jonas (2021-11-07), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, arXiv:2103.14749
  7. ^ Richie, Russell; Grover, Sachin; Tsui, Fuchiang (Rich) (May 2022). Demner-Fushman, Dina; Cohen, Kevin Bretonnel; Ananiadou, Sophia; Tsujii, Junichi (eds.). "Inter-annotator agreement is not the ceiling of machine learning performance: Evidence from a comprehensive set of simulations". Proceedings of the 21st Workshop on Biomedical Language Processing. Dublin, Ireland: Association for Computational Linguistics: 275–284. doi:10.18653/v1/2022.bionlp-1.26.
  8. ^ Artstein, Ron (2017), Ide, Nancy; Pustejovsky, James (eds.), "Inter-annotator Agreement", Handbook of Linguistic Annotation, Dordrecht: Springer Netherlands, pp. 297–313, doi:10.1007/978-94-024-0881-2_11, ISBN 978-94-024-0881-2, retrieved 2025-02-22
  9. ^ Nie, Yixin; Zhou, Xiang; Bansal, Mohit (November 2020). "What Can We Learn from Collective Human Opinions on Natural Language Inference Data?". In Webber, Bonnie; Cohn, Trevor; He, Yulan; Liu, Yang (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics. pp. 9131–9143. doi:10.18653/v1/2020.emnlp-main.734.
  10. ^ Pavlick, Ellie; Kwiatkowski, Tom (November 2019). "Inherent Disagreements in Human Textual Inferences". Transactions of the Association for Computational Linguistics. 7: 677–694. doi:10.1162/tacl_a_00293. ISSN 2307-387X.
  11. ^ Gururangan, Suchin; Swayamdipta, Swabha; Levy, Omer; Schwartz, Roy; Bowman, Samuel R.; Smith, Noah A. (2018-04-16), Annotation Artifacts in Natural Language Inference Data, arXiv:1803.02324
  12. ^ Deng, Chunyuan; Zhao, Yilun; Tang, Xiangru; Gerstein, Mark; Cohan, Arman (June 2024). "Investigating Data Contamination in Modern Benchmarks for Large Language Models". In Duh, Kevin; Gomez, Helena; Bethard, Steven (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Mexico City, Mexico: Association for Computational Linguistics. pp. 8706–8719. arXiv:2311.09783. doi:10.18653/v1/2024.naacl-long.482.
  13. ^ LI, Yanyang (2025-02-17), lyy1994/awesome-data-contamination, retrieved 2025-02-22
  14. ^ Richardson, Matthew; Burges, Christopher J.C.; Renshaw, Erin (October 2013). "MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text". In Yarowsky, David; Baldwin, Timothy; Korhonen, Anna; Livescu, Karen; Bethard, Steven (eds.). Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Seattle, Washington, USA: Association for Computational Linguistics. pp. 193–203. doi:10.18653/v1/D13-1020.
  15. ^ Rajpurkar, Pranav; Zhang, Jian; Lopyrev, Konstantin; Liang, Percy (2016-10-11), SQuAD: 100,000+ Questions for Machine Comprehension of Text, arXiv:1606.05250
  16. ^ Rajpurkar, Pranav; Jia, Robin; Liang, Percy (2018-06-11), Know What You Don't Know: Unanswerable Questions for SQuAD, arXiv:1806.03822
  17. ^ Joshi, Mandar; Choi, Eunsol; Weld, Daniel S.; Zettlemoyer, Luke (2017-05-13), TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, arXiv:1705.03551
  18. ^ Dunn, Matthew; Sagun, Levent; Higgins, Mike; Guney, V. Ugur; Cirik, Volkan; Cho, Kyunghyun (2017-06-11), SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine, arXiv:1704.05179
  19. ^ Clark, Peter; Cowhey, Isaac; Etzioni, Oren; Khot, Tushar; Sabharwal, Ashish; Schoenick, Carissa; Tafjord, Oyvind (2018-03-14), Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, arXiv:1803.05457
  20. ^ Yang, Zhilin; Qi, Peng; Zhang, Saizheng; Bengio, Yoshua; Cohen, William W.; Salakhutdinov, Ruslan; Manning, Christopher D. (2018-09-25), HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering, arXiv:1809.09600
  21. ^ Lin, Stephanie; Hilton, Jacob; Evans, Owain (2022-05-08), TruthfulQA: Measuring How Models Mimic Human Falsehoods, arXiv:2109.07958
  22. ^ Geva, Mor; Khashabi, Daniel; Segal, Elad; Khot, Tushar; Roth, Dan; Berant, Jonathan (2021-04-26). "Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies". Transactions of the Association for Computational Linguistics. 9: 346–361. doi:10.1162/tacl_a_00370. ISSN 2307-387X.
  23. ^ Levesque, Hector; Davis, Ernest; Morgenstern, Leora (2012). The Winograd Schema Challenge. Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning.
  24. ^ Kocijan, Vid; Davis, Ernest; Lukasiewicz, Thomas; Marcus, Gary; Morgenstern, Leora (2023-07-11). "The defeat of the Winograd Schema Challenge". Artificial Intelligence. 325: 103971. arXiv:2201.02387. doi:10.1016/j.artint.2023.103971. ISSN 0004-3702. S2CID 245827747.
  25. ^ Sakaguchi, Keisuke; Le Bras, Ronan; Bhagavatula, Chandra; Choi, Yejin (2019). "WinoGrande: An Adversarial Winograd Schema Challenge at Scale". arXiv:1907.10641 [cs.CL].
  26. ^ Bowman, Samuel R.; Angeli, Gabor; Potts, Christopher; Manning, Christopher D. (September 2015). "A large annotated corpus for learning natural language inference". In Màrquez, Lluís; Callison-Burch, Chris; Su, Jian (eds.). Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics. pp. 632–642. arXiv:1508.05326. doi:10.18653/v1/D15-1075.
  27. ^ "The Stanford Natural Language Processing Group". nlp.stanford.edu. Retrieved 2025-02-22.
  28. ^ Williams, Adina; Nangia, Nikita; Bowman, Samuel R. (2018-02-19), A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference, arXiv:1704.05426
  29. ^ Chen, Danqi; Bolton, Jason; Manning, Christopher D. (2016-08-08), A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task, arXiv:1606.02858
  30. ^ Zellers, Rowan; Bisk, Yonatan; Schwartz, Roy; Choi, Yejin (2018-08-16), SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference, arXiv:1808.05326
  31. ^ Zellers, Rowan; Holtzman, Ari; Bisk, Yonatan; Farhadi, Ali; Choi, Yejin (2019-05-19), HellaSwag: Can a Machine Really Finish Your Sentence?, arXiv:1905.07830
  32. ^ "HellaSwag". rowanzellers.com. Retrieved 2025-02-06.
  33. ^ Lai, Guokun; Xie, Qizhe; Liu, Hanxiao; Yang, Yiming; Hovy, Eduard (2017-12-05), RACE: Large-scale ReAding Comprehension Dataset From Examinations, arXiv:1704.04683
  34. ^ Paperno, Denis; Kruszewski, Germán; Lazaridou, Angeliki; Pham, Quan Ngoc; Bernardi, Raffaella; Pezzelle, Sandro; Baroni, Marco; Boleda, Gemma; Fernández, Raquel (2016-06-20), The LAMBADA dataset: Word prediction requiring a broad discourse context, arXiv:1606.06031
  35. ^ Wang, Alex; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2018). "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding". arXiv:1804.07461 [cs.CL].
  36. ^ "GLUE Benchmark". gluebenchmark.com. Retrieved 2019-02-25.
  37. ^ Wang, Alex; Pruksachatkun, Yada; Nangia, Nikita; Singh, Amanpreet; Michael, Julian; Hill, Felix; Levy, Omer; Bowman, Samuel R. (2020-02-13), SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems, arXiv:1905.00537
  38. ^ Srivastava, Aarohi; Rastogi, Abhinav; Rao, Abhishek; Shoeb, Abu Awal Md; Abid, Abubakar; Fisch, Adam; Brown, Adam R.; Santoro, Adam; Gupta, Aditya (2023-06-12), Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, arXiv:2206.04615
  39. ^ Suzgun, Mirac; Scales, Nathan; Schärli, Nathanael; Gehrmann, Sebastian; Tay, Yi; Chung, Hyung Won; Chowdhery, Aakanksha; Le, Quoc V.; Chi, Ed H. (2022-10-17), Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them, arXiv:2210.09261
  40. ^ Mialon, Grégoire; Fourrier, Clémentine; Swift, Craig; Wolf, Thomas; LeCun, Yann; Scialom, Thomas (2023-11-21), GAIA: a benchmark for General AI Assistants, arXiv:2311.12983
  41. ^ Zhou, Shuyan; Xu, Frank F.; Zhu, Hao; Zhou, Xuhui; Lo, Robert; Sridhar, Abishek; Cheng, Xianyi; Ou, Tianyue; Bisk, Yonatan (2024-04-16), WebArena: A Realistic Web Environment for Building Autonomous Agents, arXiv:2307.13854
  42. ^ Tay, Yi; Dehghani, Mostafa; Abnar, Samira; Shen, Yikang; Bahri, Dara; Pham, Philip; Rao, Jinfeng; Yang, Liu; Ruder, Sebastian (2020-11-08), Long Range Arena: A Benchmark for Efficient Transformers, arXiv:2011.04006
  43. ^ https://x.com/GregKamradt/status/1722386725635580292
  44. ^ An, Chenxin; Gong, Shansan; Zhong, Ming; Zhao, Xingjian; Li, Mukai; Zhang, Jun; Kong, Lingpeng; Qiu, Xipeng (August 2024). Ku, Lun-Wei; Martins, Andre; Srikumar, Vivek (eds.). "L-Eval: Instituting Standardized Evaluation for Long Context Language Models". Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: Association for Computational Linguistics: 14388–14411. arXiv:2307.11088. doi:10.18653/v1/2024.acl-long.776.
  45. ^ Zhang, Xinrong; Chen, Yingfa; Hu, Shengding; Xu, Zihang; Chen, Junhao; Hao, Moo Khai; Han, Xu; Thai, Zhen Leng; Wang, Shuo (2024-02-24), ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens, arXiv:2402.13718
  46. ^ Shaham, Uri; Ivgi, Maor; Efrat, Avia; Berant, Jonathan; Levy, Omer (2023-12-17), ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding, arXiv:2305.14196
  47. ^ Li, Tianle; Zhang, Ge; Do, Quy Duc; Yue, Xiang; Chen, Wenhu (2024-06-12), Long-context LLMs Struggle with Long In-context Learning, arXiv:2404.02060
  48. ^ "LongBench v2". longbench2.github.io. Retrieved 2025-02-21.
  49. ^ Bai, Yushi; Tu, Shangqing; Zhang, Jiajie; Peng, Hao; Wang, Xiaozhi; Lv, Xin; Cao, Shulin; Xu, Jiazheng; Hou, Lei (2025-01-03), LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks, arXiv:2412.15204
  50. ^ Hsieh, Cheng-Ping; Sun, Simeng; Kriman, Samuel; Acharya, Shantanu; Rekesh, Dima; Jia, Fei; Zhang, Yang; Ginsburg, Boris (2024-08-06), RULER: What's the Real Context Size of Your Long-Context Language Models?, arXiv:2404.06654
  51. ^ Lee, Jinhyuk; Chen, Anthony; Dai, Zhuyun; Dua, Dheeru; Sachan, Devendra Singh; Boratko, Michael; Luan, Yi; Arnold, Sébastien M. R.; Perot, Vincent (2024-06-19), Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?, arXiv:2406.13121
  52. ^ Kushman, Nate; Artzi, Yoav; Zettlemoyer, Luke; Barzilay, Regina (June 2014). Toutanova, Kristina; Wu, Hua (eds.). "Learning to Automatically Solve Algebra Word Problems". Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Baltimore, Maryland: Association for Computational Linguistics: 271–281. doi:10.3115/v1/P14-1026.
  53. ^ Huang, Danqing; Shi, Shuming; Lin, Chin-Yew; Yin, Jian; Ma, Wei-Ying (August 2016). Erk, Katrin; Smith, Noah A. (eds.). "How well do Computers Solve Math Word Problems? Large-Scale Dataset Construction and Evaluation". Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany: Association for Computational Linguistics: 887–896. doi:10.18653/v1/P16-1084.
  54. ^ Wang, Yan; Liu, Xiaojiang; Shi, Shuming (September 2017). "Deep Neural Solver for Math Word Problems". In Palmer, Martha; Hwa, Rebecca; Riedel, Sebastian (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics. pp. 845–854. doi:10.18653/v1/D17-1088.
  55. ^ Ling, Wang; Yogatama, Dani; Dyer, Chris; Blunsom, Phil (July 2017). Barzilay, Regina; Kan, Min-Yen (eds.). "Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems". Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics: 158–167. doi:10.18653/v1/P17-1015.
  56. ^ Cobbe, Karl; Kosaraju, Vineet; Bavarian, Mohammad; Chen, Mark; Jun, Heewoo; Kaiser, Lukasz; Plappert, Matthias; Tworek, Jerry; Hilton, Jacob (2021-11-18), Training Verifiers to Solve Math Word Problems, arXiv:2110.14168
  57. ^ Hendrycks, Dan; Burns, Collin; Basart, Steven; Zou, Andy; Mazeika, Mantas; Song, Dawn; Steinhardt, Jacob (2021-01-12), Measuring Massive Multitask Language Understanding, arXiv:2009.03300
  58. ^ Wang, Yubo; Ma, Xueguang; Zhang, Ge; Ni, Yuansheng; Chandra, Abhranil; Guo, Shiguang; Ren, Weiming; Arulraj, Aaran; He, Xuan (2024-11-06), MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark, arXiv:2406.01574
  59. ^ Hendrycks, Dan; Burns, Collin; Kadavath, Saurav; Arora, Akul; Basart, Steven; Tang, Eric; Song, Dawn; Steinhardt, Jacob (2021-11-08), Measuring Mathematical Problem Solving With the MATH Dataset, arXiv:2103.03874
  60. ^ Amini, Aida; Gabriel, Saadia; Lin, Peter; Koncel-Kedziorski, Rik; Choi, Yejin; Hajishirzi, Hannaneh (2019), MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms, arXiv:1905.13319
  61. ^ a b Austin, Jacob; Odena, Augustus; Nye, Maxwell; Bosma, Maarten; Michalewski, Henryk; Dohan, David; Jiang, Ellen; Cai, Carrie; Terry, Michael (2021-08-16), Program Synthesis with Large Language Models, arXiv:2108.07732
  62. ^ math-eval (2025-01-26), math-eval/MathEval, retrieved 2025-01-27
  63. ^ Chen, Wenhu; Yin, Ming; Ku, Max; Lu, Pan; Wan, Yixin; Ma, Xueguang; Xu, Jianyu; Wang, Xinyi; Xia, Tony (December 2023). "TheoremQA: A Theorem-driven Question Answering Dataset". In Bouamor, Houda; Pino, Juan; Bali, Kalika (eds.). Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics. pp. 7889–7901. arXiv:2305.12524. doi:10.18653/v1/2023.emnlp-main.489.
  64. ^ openai/miniF2F, OpenAI, 2025-02-01, retrieved 2025-02-03
  65. ^ Gao, Bofei; Song, Feifan; Yang, Zhe; Cai, Zefan; Miao, Yibo; Dong, Qingxiu; Li, Lei; Ma, Chenghao; Chen, Liang (2024-12-24), Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models, arXiv:2410.07985
  66. ^ Glazer, Elliot; Erdil, Ege; Besiroglu, Tamay; Chicharro, Diego; Chen, Evan; Gunning, Alex; Olsson, Caroline Falkman; Denain, Jean-Stanislas; Ho, Anson (2024-12-20), FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI, arXiv:2411.04872
  67. ^ "MathArena.ai". matharena.ai. Retrieved 2025-02-22.
  68. ^ Hendrycks, Dan; Basart, Steven; Kadavath, Saurav; Mazeika, Mantas; Arora, Akul; Guo, Ethan; Burns, Collin; Puranik, Samir; He, Horace (2021-11-08), Measuring Coding Challenge Competence With APPS, arXiv:2105.09938
  69. ^ Chen, Mark; Tworek, Jerry; Jun, Heewoo; Yuan, Qiming; Pinto, Henrique Ponde de Oliveira; Kaplan, Jared; Edwards, Harri; Burda, Yuri; Joseph, Nicholas (2021-07-14), Evaluating Large Language Models Trained on Code, arXiv:2107.03374
  70. ^ "CodeElo". codeelo-bench.github.io. Retrieved 2025-02-13.
  71. ^ Jimenez, Carlos E.; Yang, John; Wettig, Alexander; Yao, Shunyu; Pei, Kexin; Press, Ofir; Narasimhan, Karthik (2024-11-11), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv:2310.06770
  72. ^ "Introducing SWE-bench Verified". openai.com.
  73. ^ "SWE-bench". www.swebench.com. Retrieved 2025-02-11.
  74. ^ openai/SWELancer-Benchmark, OpenAI, 2025-02-21, retrieved 2025-02-21
  75. ^ Miserendino, Samuel; Wang, Michele; Patwardhan, Tejal; Heidecke, Johannes (2025-02-19), SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?, arXiv:2502.12115
  76. ^ Ouyang, Anne; Guo, Simon; Arora, Simran; Zhang, Alex L.; Hu, William; Ré, Christopher; Mirhoseini, Azalia (2025-02-18), KernelBench: Can LLMs Write Efficient GPU Kernels?, arXiv:2502.10517
  77. ^ Rein, David; Hou, Betty Li; Stickland, Asa Cooper; Petty, Jackson; Pang, Richard Yuanzhe; Dirani, Julien; Michael, Julian; Bowman, Samuel R. (2023-11-20), GPQA: A Graduate-Level Google-Proof Q&A Benchmark, arXiv:2311.12022
  78. ^ Team, M.-A.-P.; Du, Xinrun; Yao, Yifan; Ma, Kaijing; Wang, Bingli; Zheng, Tianyu; Zhu, Kang; Liu, Minghao; Liang, Yiming (2025-02-20), SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines, arXiv:2502.14739
  79. ^ Cui, Ruixiang (2025-02-03), ruixiangcui/AGIEval, retrieved 2025-02-03
  80. ^ "OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI". gair-nlp.github.io. Retrieved 2025-02-03.
  81. ^ He, Chaoqun; Luo, Renjie; Bai, Yuzhuo; Hu, Shengding; Thai, Zhen Leng; Shen, Junhao; Hu, Jinyi; Han, Xu; Huang, Yujie (2024-06-06), OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems, arXiv:2402.14008
  82. ^ "ARC Prize". ARC Prize. Retrieved 2025-01-27.
  83. ^ "LiveBench". livebench.ai. Retrieved 2025-01-27.
  84. ^ "Humanity's Last Exam". lastexam.ai. Retrieved 2025-02-02.
Retrieved from "http://en.wiki.x.io/w/index.php?title=Language_model_benchmark&oldid=1277368126"