BenchRisk Scores

Score Details

These scores are all scoped around benchmarks intended to score information-only chatbots. Agentic systems (i.e., those taking direct action) are not in scope for this release versioned as BenchRisk-ChatBot-v1.0.

Search scores
X Clear Filter

• Scored on

  April 1, 2025

  71 adopted mitigations

  minimum score of 51

  AILuminate10

  Benchmark indicates the propensity of AI systems to respond in a hazardous manner to prompts from malicious or vulnerable users that might result in harm to themselves or others

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • lower risk of information degrading through time.
  • lower risk of statistically biased results misleading.
  • lower risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  75 percent of known longevity risk mitigated

  77 percent of known correctness risk mitigated

  86 percent of known intelligibility risk mitigated

  51 percent of known comprehensiveness risk mitigated

  80 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  58 adopted mitigations

  minimum score of 44

  ARCAGIPrivate

  Benchmark indicates how well a system can perform abstract reasoning tasks that require generalization from minimal examples, reflecting human-like fluid intelligence.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • moderate risk of information degrading through time.
  • lower risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  52 percent of known longevity risk mitigated

  83 percent of known correctness risk mitigated

  68 percent of known intelligibility risk mitigated

  44 percent of known comprehensiveness risk mitigated

  62 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  49 adopted mitigations

  minimum score of 10

  ARCAGIPublic

  Benchmark indicates how well a system can perform abstract reasoning tasks that require generalization from minimal examples, reflecting human-like fluid intelligence.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • lower risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  10 percent of known longevity risk mitigated

  74 percent of known correctness risk mitigated

  63 percent of known intelligibility risk mitigated

  44 percent of known comprehensiveness risk mitigated

  62 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  51 adopted mitigations

  minimum score of 5

  WinoGrande

  Benchmark indicates how well a system can answer 44k fill-in-a-blank questions with binary options associated with commonsense reasoning

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • moderate risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  69 percent of known correctness risk mitigated

  66 percent of known intelligibility risk mitigated

  32 percent of known comprehensiveness risk mitigated

  86 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  44 adopted mitigations

  minimum score of 6

  HumanEval

  Benchmark indicates how well the system under test can support a human writing python code by specifying docstrings whose function bodies are completed by the system.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • lower risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  6 percent of known longevity risk mitigated

  71 percent of known correctness risk mitigated

  40 percent of known intelligibility risk mitigated

  46 percent of known comprehensiveness risk mitigated

  82 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  41 adopted mitigations

  minimum score of 5

  GPQA

  Benchmark indicates how well a system can answer graduate-level multiple-choice questions in biology, chemistry, and physics, with performance comparable to skilled non-experts even when provided unrestricted internet access.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • lower risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  70 percent of known correctness risk mitigated

  60 percent of known intelligibility risk mitigated

  41 percent of known comprehensiveness risk mitigated

  36 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  48 adopted mitigations

  minimum score of 10

  Toxigen

  Benchmark indicates whether a system detects implicit hate speech targeting minority groups while avoiding false alarms

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • moderate risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  10 percent of known longevity risk mitigated

  52 percent of known correctness risk mitigated

  67 percent of known intelligibility risk mitigated

  34 percent of known comprehensiveness risk mitigated

  82 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  41 adopted mitigations

  minimum score of 0

  BBQ

  Benchmark measures social biases against people belonging to protected classes along nine social dimensions relevant for U.S. English-speaking contexts.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • moderate risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  0 percent of known longevity risk mitigated

  66 percent of known correctness risk mitigated

  47 percent of known intelligibility risk mitigated

  51 percent of known comprehensiveness risk mitigated

  80 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  44 adopted mitigations

  minimum score of 23

  BigBenchHard

  Benchmark indicates how well a system can perform 23 challenging tasks from the BIG-Bench Hard (BBH) suite, assessing its ability to solve complex reasoning problems

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  28 percent of known longevity risk mitigated

  48 percent of known correctness risk mitigated

  44 percent of known intelligibility risk mitigated

  23 percent of known comprehensiveness risk mitigated

  64 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  43 adopted mitigations

  minimum score of 23

  BigBenchExtraHard

  Benchmark indicates how well a system can perform 23 novel tasks that probe advanced reasoning capabilities, each designed to be significantly more challenging than its counterpart in the original BIG-Bench (BBH) suite.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  28 percent of known longevity risk mitigated

  48 percent of known correctness risk mitigated

  38 percent of known intelligibility risk mitigated

  23 percent of known comprehensiveness risk mitigated

  64 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  39 adopted mitigations

  minimum score of 21

  AILuminate05

  Benchmark indicates the propensity of AI systems to respond in a hazardous manner to prompts from malicious or vulnerable users that might result in harm to themselves or others.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  21 percent of known longevity risk mitigated

  35 percent of known correctness risk mitigated

  66 percent of known intelligibility risk mitigated

  48 percent of known comprehensiveness risk mitigated

  37 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  41 adopted mitigations

  minimum score of 11

  HumanitysLastExam

  Benchmark indicates whether a model can answer expert-level academic questions across disciplines with precision and understanding, serving as a key signal of the model’s reliability for users who depend on it for rigorous, high-stakes reasoning.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  11 percent of known longevity risk mitigated

  40 percent of known correctness risk mitigated

  61 percent of known intelligibility risk mitigated

  52 percent of known comprehensiveness risk mitigated

  37 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  40 adopted mitigations

  minimum score of 22

  BigBench

  Benchmark indicates how well a system can perform across a diverse set of 204 challenging tasks spanning linguistics, mathematics, commonsense reasoning, and more, reflecting its generalization and reasoning capabilities.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  22 percent of known longevity risk mitigated

  41 percent of known correctness risk mitigated

  38 percent of known intelligibility risk mitigated

  23 percent of known comprehensiveness risk mitigated

  44 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  31 adopted mitigations

  minimum score of 0

  Ethics

  Benchmark measures model capacity for reflecting human notions of justice, well-being, duties, virtues, and commonsense morality

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  0 percent of known longevity risk mitigated

  48 percent of known correctness risk mitigated

  44 percent of known intelligibility risk mitigated

  47 percent of known comprehensiveness risk mitigated

  46 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  30 adopted mitigations

  minimum score of 0

  TruthfulQA

  Benchmark indicates whether a model can generate factually accurate answers to questions while avoiding common human misconceptions, measuring its susceptibility to imitative falsehoods—critical for users who depend on AI systems for reliable information in high-stakes domains like medicine, law, and science.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • lower risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  0 percent of known longevity risk mitigated

  42 percent of known correctness risk mitigated

  37 percent of known intelligibility risk mitigated

  30 percent of known comprehensiveness risk mitigated

  78 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  38 adopted mitigations

  minimum score of 5

  HellaSwag

  Benchmark capacity to reason about physical and social concepts by prompting for sentence endings from challenging, adversarially filtered alternatives, providing users with a measure of the scope of model capacity.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  42 percent of known correctness risk mitigated

  33 percent of known intelligibility risk mitigated

  58 percent of known comprehensiveness risk mitigated

  68 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  36 adopted mitigations

  minimum score of 11

  MMLU

  Benchmark evaluates a language model’s multitask academic and professional knowledge by measuring its accuracy on multiple-choice questions across 57 diverse subjects.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  11 percent of known longevity risk mitigated

  35 percent of known correctness risk mitigated

  40 percent of known intelligibility risk mitigated

  50 percent of known comprehensiveness risk mitigated

  62 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  30 adopted mitigations

  minimum score of 5

  GSM8K

  Benchmark measures model capability in solving diverse grade school math word problems.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  42 percent of known correctness risk mitigated

  34 percent of known intelligibility risk mitigated

  16 percent of known comprehensiveness risk mitigated

  60 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  39 adopted mitigations

  minimum score of 5

  DecodingTrustToxicity

  Benchmark evaluates the toxicity of sentence completions.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • moderate risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  34 percent of known correctness risk mitigated

  48 percent of known intelligibility risk mitigated

  56 percent of known comprehensiveness risk mitigated

  57 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  30 adopted mitigations

  minimum score of 5

  AIRBench

  Benchmark evaluates LLM safety by measuring responses to prompts that are mapped to granular risk categories derived from government regulations and AI policies.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  33 percent of known correctness risk mitigated

  46 percent of known intelligibility risk mitigated

  42 percent of known comprehensiveness risk mitigated

  42 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  29 adopted mitigations

  minimum score of 5

  AnthropicRedTeam

  Benchmark indicates whether an LLM will refuse to generate malicious code when prompted, signaling its resistance to abuse and helping users evaluate the model’s safety and robustness.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • moderate risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  5 percent of known longevity risk mitigated

  28 percent of known correctness risk mitigated

  56 percent of known intelligibility risk mitigated

  25 percent of known comprehensiveness risk mitigated

  37 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  33 adopted mitigations

  minimum score of 7

  BOLDBias

  Benchmark indicates whether an open-ended text generation model maintains fairness and avoids social biases across domains such as profession, gender, race, religion, and ideology when prompted with real-world English contexts, helping users evaluate the model’s equity and ethical behavior.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • high risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  7 percent of known longevity risk mitigated

  16 percent of known correctness risk mitigated

  36 percent of known intelligibility risk mitigated

  47 percent of known comprehensiveness risk mitigated

  21 percent of known consistency risk mitigated

• Scored on

  April 1, 2025

  23 adopted mitigations

  minimum score of 0

  RealToxicityPrompts

  Benchmark measures a language model's propensity for toxic degeneration, quantifying how even seemingly innocuous, naturally-occurring prompts can trigger the generation of toxic text.

  Refer to the original reference for more details about the benchmark

  
  The benchmark presents a...

  • high risk of information degrading through time.
  • high risk of statistically biased results misleading.
  • high risk of misunderstanding what the benchmark evidences.
  • moderate risk of circumstance not being covered when the benchmark may reasonably be expected to cover the circumstance.
  • high risk of randomness misleading via scores not representative of the system.

  Numerically, this is supported by the following scores:

  0 percent of known longevity risk mitigated

  12 percent of known correctness risk mitigated

  36 percent of known intelligibility risk mitigated

  58 percent of known comprehensiveness risk mitigated

  20 percent of known consistency risk mitigated