CaltechAUTHORS
Published January 29, 2026 | Version Published
Journal Article Open

A benchmark of expert-level academic questions to assess AI capabilities

Creators

  • 1. Center for AI Safety, San Francisco CA, USA
  • 2. Scale AI, San Francisco CA, USA
  • 3. Independent Researcher
  • 4. ROR icon Brown University
  • 5. ROR icon McGill University
  • 6. ROR icon Institute of Mathematics
  • 7. Kiev School of Economics, Kyiv, Ukraine
  • 8. ROR icon Carnegie Mellon University
  • 9. ROR icon RWTH Aachen University
  • 10. ROR icon University of Cambridge
  • 11. ROR icon National Technical University of Ukraine "Igor Sikorsky Kyiv Polytechnic Institute"
  • 12. ROR icon Queen's University
  • 13. ROR icon Stanford University
  • 14. ROR icon University of Washington
  • 15. ROR icon University of California, San Diego
  • 16. ROR icon University of Porto
  • 17. ROR icon Eötvös Loránd University
  • 18. Nimbus AI, Honolulu, HI, USA
  • 19. ROR icon ETH Zurich
  • 20. ROR icon Durham University
  • 21. ROR icon Georgia Southern University
  • 22. ROR icon University of Minnesota
  • 23. ROR icon Queen Mary University of London
  • 24. ROR icon Microsoft (United States)
  • 25. ROR icon Auckland University of Technology
  • 26. ROR icon Alberta Health Services
  • 27. ROR icon University of Illinois at Chicago
  • 28. Hereford College of Arts, Hereford, UK
  • 29. ROR icon Princeton University
  • 30. ROR icon University of Canterbury
  • 31. ROR icon Metropolitan State University of Denver
  • 32. ROR icon Massachusetts Institute of Technology
  • 33. Accenture Labs, Washington, DC, USA
  • 34. ROR icon Tufts University
  • 35. ROR icon Jackson Laboratory
  • 36. ROR icon French Institute for Research in Computer Science and Automation
  • 37. ROR icon University of California, Berkeley
  • 38. ROR icon University of British Columbia
  • 39. ROR icon Ross University School of Medicine
  • 40. ROR icon École Polytechnique Fédérale de Lausanne
  • 41. ROR icon Concordia University
  • 42. ROR icon Institute of Science and Technology Austria
  • 43. ROR icon National University of Singapore
  • 44. ROR icon California Institute of Technology
  • 45. ROR icon University of Oxford
  • 46. ROR icon Universidade de São Paulo
  • 47. ROR icon Humboldt-Universität zu Berlin
  • 48. ROR icon Charité - University Medicine Berlin
  • 49. ROR icon Columbia University
  • 50. ROR icon University of Southern California
  • 51. ROR icon Stony Brook University
  • 52. ROR icon University of Luxembourg
  • 53. ROR icon Universidade Federal de Juiz de Fora
  • 54. ROR icon Rockwell Automation (United States)
  • 55. Contramont Research, San Francisco, CA, USA
  • 56. ROR icon École Normale Supérieure - PSL
  • 57. ROR icon Sorbonne University
  • 58. ROR icon University of Toronto
  • 59. ROR icon University of Tübingen
  • 60. ROR icon Sapienza University of Rome
  • 61. ROR icon University of North Texas
  • 62. ROR icon Institut Polytechnique de Paris
  • 63. ROR icon University of the Philippines Manila
  • 64. ROR icon University of Bath
  • 65. ROR icon Maastricht University
  • 66. ROR icon Washington University in St. Louis
  • 67. ROR icon University of California, Los Angeles
  • 68. ROR icon University of Paris-Saclay
  • 69. ROR icon French National Centre for Scientific Research
  • 70. ROR icon Martin Luther University Halle-Wittenberg
  • 71. ROR icon Leibniz University Hannover
  • 72. Diverging Mathematics, Boston, MA, USA
  • 73. ROR icon Indian Institute of Technology Bombay
  • 74. ROR icon Institute for Molecular Manufacturing
  • 75. ROR icon University of Michigan–Ann Arbor
  • 76. ROR icon DeepMind (United Kingdom)
  • 77. ROR icon Vrije Universiteit Brussel
  • 78. ROR icon Universitair Ziekenhuis Brussel
  • 79. ROR icon PeopleTec (United States)
  • 80. ROR icon University of Chicago
  • 81. ROR icon Technion – Israel Institute of Technology
  • 82. ROR icon University of Miami
  • 83. ROR icon Technical University of Berlin
  • 84. ROR icon University of Manchester
  • 85. ROR icon University of Illinois Urbana-Champaign
  • 86. ROR icon University of Calgary
  • 87. ROR icon Ibero American University
  • 88. ROR icon TU Wien
  • 89. ROR icon University of Wisconsin–Madison
  • 90. ROR icon Yale University
  • 91. ROR icon University of Edinburgh
  • 92. ROR icon École Normale Supérieure Paris-Saclay
  • 93. ROR icon University of Western Australia
  • 94. Snorkel AI, Redwood City, CA, USA
  • 95. ROR icon New York University
  • 96. ROR icon Mohamed bin Zayed University of Artificial Intelligence
  • 97. ROR icon University of Waterloo
  • 98. ROR icon University of Maryland, College Park
  • 99. ROR icon Manhattan School of Music
  • 100. ROR icon Leiden University
  • 101. Synbionix, Casselberry, FL, USA
  • 102. ROR icon The Open University
  • 103. Corteva Agriscience, Indianapolis, IN, USA
  • 104. ROR icon Sanford Burnham Prebys Medical Discovery Institute
  • 105. ROR icon Yonsei University
  • 106. ROR icon Harvard University
  • 107. ROR icon Cornell University
  • 108. ROR icon University of Leeds
  • 109. ROR icon Arizona State University
  • 110. ROR icon Swinburne University of Technology
  • 111. ROR icon KU Leuven
  • 112. ROR icon St. Petersburg College
  • 113. ROR icon National Agrarian University
  • 114. ROR icon Brandenburg University of Technology
  • 115. INSAIT, Sofia, Bulgaria
  • 116. ROR icon Ruhr University Bochum
  • 117. ROR icon National Information Processing Institute
  • 118. ROR icon Charles University
  • 119. ROR icon Cranfield University
  • 120. ROR icon University of Copenhagen
  • 121. TRR Designs, Fayetteville, AR, USA
  • 122. ROR icon University of Sydney
  • 123. ROR icon Indian Institute of Technology Delhi
  • 124. ROR icon University of Buenos Aires
  • 125. ROR icon University of Technology Sydney
  • 126. ROR icon Indiana State University
  • 127. ROR icon Australian National University
  • 128. ROR icon Royal Institute of Technology
  • 129. ROR icon University of Amsterdam
  • 130. ROR icon Ben-Gurion University of the Negev
  • 131. ROR icon Donald & Barbara Zucker School of Medicine at Hofstra/Northwell
  • 132. Cohere, Toronto Ontario, Canada
  • 133. Siili Solutions, Helsinki, Finland
  • 134. ROR icon University of Pennsylvania
  • 135. ROR icon Aalto University
  • 136. ROR icon Toyota Technological Institute at Chicago
  • 137. ROR icon Northeastern University
  • 138. ROR icon Case Western Reserve University
  • 139. ROR icon University of Windsor
  • 140. ROR icon St. Jude Children's Research Hospital
  • 141. ROR icon Rochester Institute of Technology
  • 142. ROR icon Emory University
  • 143. Anthropic, San Francisco, CA, USA
  • 144. ROR icon European Organization for Nuclear Research
  • 145. ROR icon University of California, Santa Barbara
  • 146. ROR icon Warsaw University of Technology
  • 147. ROR icon Hewlett Packard Enterprise (United States)
  • 148. ROR icon North Carolina State University
  • 149. ROR icon University of Houston
  • 150. ROR icon All India Institute of Medical Sciences
  • 151. ROR icon Tel Aviv University
  • 152. ROR icon Georgia Institute of Technology
  • 153. ROR icon Northwestern University
  • 154. ROR icon University of Arizona
  • 155. ROR icon University of Lisbon
  • 156. ROR icon Indian Institute of Technology Kharagpur
  • 157. ROR icon Posts and Telecommunications Institute of Technology - Ho Chi Minh City
  • 158. ROR icon Duke University
  • 159. ROR icon Mila - Quebec Artificial Intelligence Institute
  • 160. ROR icon University College London
  • 161. ROR icon University of Zurich
  • 162. UK AI Safety Institute, London, UK
  • 163. ROR icon University of Padua
  • 164. ROR icon Boston University
  • 165. ROR icon Royal Veterinary College
  • 166. SDAIA, Riyadh, Saudi Arabia
  • 167. ROR icon The Ohio State University
  • 168. ROR icon University of Montreal
  • 169. ROR icon Cairo University
  • 170. ROR icon University of Valencia
  • 171. ROR icon Monash University
  • 172. ROR icon Van Andel Institute
  • 173. ROR icon Larkin Community Hospital
  • 174. ROR icon The University of Texas at Dallas
  • 175. ROR icon Canadian University of Dubai
  • 176. ROR icon Hebrew University of Jerusalem
  • 177. ROR icon University of Milano-Bicocca
  • 178. ROR icon University of Massachusetts Lowell
  • 179. ROR icon Virginia Tech
  • 180. ROR icon University of Geneva
  • 181. ROR icon Google (United States)
  • 182. ROR icon California Polytechnic State University
  • 183. ROR icon Alexandru Ioan Cuza University
  • 184. ROR icon University of Mannheim
  • 185. ROR icon Stockholm University
  • 186. College of Eastern Idaho, Idaho Falls, ID, USA
  • 187. Intrinsic Innovation, Mountain View, CA, USA
  • 188. Ivy Natal, San Francisco, CA, USA
  • 189. ROR icon King Saud University
  • 190. SAMPE Switzerland, Zurich, Switzerland
  • 191. CERo Therapeutics Holdings, South San Francisco, CA, USA
  • 192. ROR icon University of Tennessee at Knoxville
  • 193. Gray Swan AI, Pittsburgh, PA, USA
  • 194. EleutherAI, Washington, DC, USA
  • 195. ROR icon Johns Hopkins University
  • 196. ROR icon University of Montpellier
  • 197. ROR icon Fraunhofer Research Institution for Marine Biotechnology and Cell Technology
  • 198. HomeEquity Bank, Toronto Ontario, Canada
  • 199. Materials Platform for Data Science, Tallinn, Estonia
  • 200. ROR icon University of Pisa
  • 201. ROR icon Georgia State University
  • 202. ROR icon Polytechnic University of the Philippines
  • 203. ROR icon University of Oregon
  • 204. ROR icon Drexel University
  • 205. ROR icon University of Mumbai
  • 206. ROR icon Gakushuin University
  • 207. ROR icon University of Guelph
  • 208. ROR icon Intuit (United States)
  • 209. CTTC / CERCA, Castelldefels, Spain
  • 210. Dyno Therapeutics, Watertown, MA, USA
  • 211. ROR icon Hospital for Sick Children
  • 212. ROR icon Temple University
  • 213. ROR icon Saint Mary's University
  • 214. ROR icon Cisco Systems (United States)
  • 215. ROR icon Indian Institute of Technology BHU
  • 216. AIM Intelligence, Seoul, South Korea
  • 217. ROR icon Seoul National University
  • 218. ROR icon The University of Texas at Arlington
  • 219. ROR icon Science and Technology Facilities Council
  • 220. ROR icon University of Vienna
  • 221. POLITEHNICA Bucharest National University of Science and Technology, Bucharest, Romania
  • 222. Abacus.AI, San Francisco, CA, USA
  • 223. ROR icon National University of Ireland, Galway
  • 224. ROR icon Eastern Institute of Technology
  • 225. ROR icon École Normale Supérieure de Lyon
  • 226. ROR icon Czech Technical University in Prague
  • 227. ROR icon Universität Hamburg
  • 228. ROR icon Helmholtz Center for Information Security
  • 229. ROR icon Universidad de Morón
  • 230. ROR icon University of Paris
  • 231. ROR icon Politecnico di Milano
  • 232. ROR icon New School
  • 233. ROR icon Max Planck Institute for Software Systems
  • 234. ROR icon OpenAI (United States)
  • 235. ROR icon University of Granada
  • 236. Modulo Research, Cambridge, UK
  • 237. ROR icon Heidelberg University
  • 238. ROR icon La Trobe University
  • 239. ROR icon Université de Yaoundé I
  • 240. ROR icon Universität Innsbruck
  • 241. Nabu Technologies, San Francisco, CA, USA
  • 242. ROR icon Chalmers University of Technology
  • 243. Unidade Local de Saúde de Lisboa Ocidental, Lisbon, Portugal
  • 244. ROR icon Children's Hospital of Orange County
  • 245. The Future Paralegals of America, New York, NY, USA
  • 246. Eastlake High School, Sammamish, WA, USA
  • 247. ROR icon Center for Scientific Research and Higher Education at Ensenada
  • 248. ROR icon University of Bradford
  • 249. ROR icon Beni-Suef University
  • 250. ROR icon Boğaziçi University
  • 251. ROR icon Mansoura University
  • 252. ROR icon University of Bristol
  • 253. ROR icon University of Oklahoma
  • 254. Jala University, Honolulu, HI, USA
  • 255. ROR icon University of Arkansas at Fayetteville
  • 256. ROR icon Florida Atlantic University
  • 257. ROR icon Bournemouth University
  • 258. ROR icon University of Warwick
  • 259. ROR icon University of Alabama in Huntsville
  • 260. ROR icon University of Hertfordshire
  • 261. OncoPrecision, New York, NY, USA
  • 262. ROR icon Central College
  • 263. ROR icon Nottingham Trent University
  • 264. ROR icon Max Planck Institute for Intelligent Systems
  • 265. ROR icon University of Virginia
  • 266. ROR icon Dartmouth College
  • 267. ROR icon Instituto de Engenharia de Sistemas e Computadores Microsistemas e Nanotecnologias
  • 268. ROR icon James Madison University
  • 269. Instituto Gonçalo Moniz, Salvador, Brazil
  • 270. ROR icon Rice University
  • 271. ROR icon Research Centre for Natural Sciences
  • 272. ROR icon Rutgers, The State University of New Jersey
  • 273. AE Studio, Marina Del Rey, CA, USA
  • 274. ROR icon Saarland University
  • 275. HUTECH, Ho Chi Minh City, Vietnam
  • 276. ROR icon Pennsylvania State University
  • 277. Intelligent Geometries, Front Royal, VA, USA
  • 278. ROR icon École Polytechnique
  • 279. ROR icon National Scientific and Technical Research Council
  • 280. ROR icon National Technological University
  • 281. John Crane UK, Slough, UK
  • 282. ROR icon The Alan Turing Institute
  • 283. ROR icon Pondicherry University
  • 284. ROR icon Leibniz Institute for Science and Mathematics Education
  • 285. ROR icon Royal Holloway University of London
  • 286. ROR icon Tanta University
  • 287. ROR icon University of Malaya
  • 288. ROR icon Hemwati Nandan Bahuguna Garhwal University
  • 289. ROR icon Mohamed I University
  • 290. LGM, Paris, France
  • 291. ROR icon Northern Illinois University
  • 292. ROR icon Bethune-Cookman University
  • 293. ROR icon University of California, Irvine
  • 294. ROR icon Central Mindanao University
  • 295. ROR icon University of the Fraser Valley
  • 296. Patched Codes, San Francisco, CA, USA
  • 297. ROR icon Missouri University of Science and Technology
  • 298. Quotient AI, Boston, MA, USA
  • 299. CSMSS Chh. Shahu College of Engineering, Aurangabad, India
  • 300. Genomia Diagnostics Research, New Delhi, India
  • 301. ROR icon Sheffield Teaching Hospitals NHS Foundation Trust
  • 302. ROR icon Forschungszentrum Jülich
  • 303. Standard Intelligence, San Francisco, CA, USA
  • 304. ROR icon RMIT University
  • 305. ROR icon German Research Centre for Artificial Intelligence
  • 306. ROR icon Fondazione Bruno Kessler
  • 307. ROR icon University of Trento
  • 308. ROR icon Chulalongkorn University
  • 309. ROR icon Aligarh Muslim University
  • 310. Happy Technologies LLC, Arlington, VA, USA
  • 311. ROR icon Menoufia University
  • 312. ROR icon Instituto Politécnico Nacional
  • 313. ROR icon University of Bologna
  • 314. ROR icon Manipal Academy of Higher Education
  • 315. ROR icon The University of Texas at Austin
  • 316. ROR icon Murdoch University
  • 317. ROR icon University of Delaware
  • 318. ROR icon Williams College
  • 319. ROR icon Perimeter Institute
  • 320. ROR icon University of Maribor
  • 321. ROR icon Brigham and Women's Hospital
  • 322. ROR icon University of Tokyo
  • 323. ROR icon Imperial College London
  • 324. ROR icon University of California, Santa Cruz
  • 325. ROR icon Vellore Institute of Technology University
  • 326. ROR icon Centre Hospitalier Universitaire de Nancy
  • 327. ROR icon Delft University of Technology
  • 328. ROR icon Scripps Research Institute
  • 329. Aleph Alpha, Heidelberg, Germany
  • 330. ROR icon George Mason University
  • 331. ROR icon Atilim University
  • 332. Leonardo Labs, Rome, Italy
  • 333. ROR icon Complexity Science Hub Vienna
  • 334. ROR icon National University of Distance Education
  • 335. ROR icon Saxion University of Applied Sciences
  • 336. ROR icon Korea Advanced Institute of Science and Technology
  • 337. ROR icon Adobe Systems (United States)
  • 338. ROR icon National Aerospace University – Kharkiv Aviation Institute
  • 339. Hexworks, Barcelona, Spain
  • 340. ROR icon Westmead Hospital
  • 341. ROR icon University of Bern
  • 342. Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau, Kaiserslautern, Germany
  • 343. SUMM AI, Munich, Germany
  • 344. ROR icon University of North Carolina at Chapel Hill
  • 345. ROR icon Konkuk University
  • 346. ROR icon University of Groningen
  • 347. ROR icon Jagiellonian University
  • 348. Minerva University, San Francisco, CA, USA
  • 349. ROR icon Aalborg University
  • 350. IBM Research, Givatayim, Israel
  • 351. ROR icon Universitat Politècnica de València
  • 352. RBC Borealis, Toronto Ontario, Canada
  • 353. Mayo Clinic, Rochester, MN, USA
  • 354. ROR icon University of Lausanne
  • 355. ROR icon Dalhousie University
  • 356. ROR icon University of Lleida
  • 357. Amazon, Seattle, WA, USA
  • 358. Dell Technologies, Round Rock, TX, USA
  • 359. ROR icon University of Seoul
  • 360. ROR icon University of Auckland
  • 361. ROR icon Morgridge Institute for Research
  • 362. ROR icon Korea University of Technology and Education
  • 363. ROR icon University of California, Davis
  • 364. ROR icon Baylor College of Medicine
  • 365. ROR icon Indraprastha Institute of Information Technology Delhi
  • 366. Two Minute Papers, Pécs, Hungary
  • 367. ADIA Lab, Abu Dhabi, UAE
  • 368. ROR icon New Jersey Institute of Technology
  • 369. ROR icon Novo Nordisk (Denmark)
  • 370. ROR icon Purdue University West Lafayette
  • 371. Gakugei Shuppan-sha, Kyoto, Japan
  • 372. ROR icon Utrecht University
  • 373. ROR icon Deutsche Telekom (Spain)
  • 374. ROR icon University of Klagenfurt
  • 375. ROR icon Max Planck Institute for Security and Privacy
  • 376. InxiteOut, Bangalore, India
  • 377. ROR icon Goethe University Frankfurt
  • 378. ROR icon University of Valle
  • 379. ROR icon Bilkent University
  • 380. Trinity School, New York, NY, USA
  • 381. ROR icon Pompeu Fabra University
  • 382. Brighton Law School, Brighton, UK
  • 383. ROR icon University of Melbourne
  • 384. ROR icon Ankara University
  • 385. ROR icon İstanbul Eğitim ve Araştırma Hastanesi
  • 386. ROR icon Austrian Institute of Technology
  • 387. ROR icon Technical University of Munich
  • 388. ROR icon Providence College
  • 389. ROR icon University of Jyväskylä
  • 390. ROR icon Weizmann Institute of Science
  • 391. ROR icon Indiana University Bloomington
  • 392. ROR icon Nanyang Technological University
  • 393. ROR icon University of Sheffield

Abstract

Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achieve more than 90% accuracy on popular benchmarks such as Measuring Massive Multitask Language Understanding1, limiting informed measurement of state-of-the-art LLM capabilities. Here, in response, we introduce Humanity’s Last Exam (HLE), a multi-modal benchmark at the frontier of human knowledge, designed to be an expert-level closed-ended academic benchmark with broad subject coverage. HLE consists of 2,500 questions across dozens of subjects, including mathematics, humanities and the natural sciences. HLE is developed globally by subject-matter experts and consists of multiple-choice and short-answer questions suitable for automated grading. Each question has a known solution that is unambiguous and easily verifiable but cannot be quickly answered by internet retrieval. State-of-the-art LLMs demonstrate low accuracy and calibration on HLE, highlighting a marked gap between current LLM capabilities and the expert human frontier on closed-ended academic questions. To inform research and policymaking upon a clear understanding of model capabilities, we publicly release HLE at https://lastexam.ai.

Copyright and License

© 2026, The Author(s). Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

Acknowledgement

The research is supported by Center for AI Safety and Scale AI.

Data Availability

The HLE dataset is open-source and available at https://huggingface.co/datasets/cais/hle. Important updates to the project and dataset will be announced at https://lastexam.ai.

Code Availability

The inference script for benchmarking AI systems on HLE is available at GitHub (https://github.com/centerforaisafety/hle).

Supplemental Material

See attached - Supplementary Information containing the contributor guidelines and human review instructions.

Files

s41586-025-09962-4.pdf

Files (4.4 MB)

Name Size
md5:cc01d95d8cd731b2d2101dd34d832f51
968.0 kB Preview Download
md5:64d43cd5066ea2e27605ae72e34dff47
3.5 MB Preview Download

Additional details

Identifiers

PMCID
PMC12851929
PMID
41606155

Related works

Describes
Journal Article: https://rdcu.be/e2mOP (URL)
Is supplemented by
Dataset: https://huggingface.co/datasets/cais/hle (URL)

Dates

Available
2026-01-28
Version of record

Caltech Custom Metadata

Caltech groups
Division of Engineering and Applied Science (EAS) , Division of Physics, Mathematics and Astronomy (PMA) , Mathematics Department , Physics Department
Publication Status
Published