Mitigations

Do you clearly, publicly, and prominently state the user information foraging task associated with your benchmark evaluation—using plain and concise language that is accessible to the intended users? By 'information foraging task,' we mean the specific goal or question that a user is trying to answer or learn about when engaging with the benchmark, reflecting their real-world information needs.

This mitigates Failure Mode 1: The information provided by the benchmark does not match with the information the benchmark user believes is provided

Affirming Benchmarks:

Do you contractually prohibit the use of large language models (LLMs) in the production of test data for your benchmark?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Do you record the specification of the LLM used for every prompt that was produced with the aid of an LLM?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Do you run a study on any system under test (SUT) that may have been privileged during prompt generation, and compare its performance to SUTs not involved in prompt generation? If an unfair advantage is found, do you drop the LLM-generated instances?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Do you avoid sourcing data from publicly available information when constructing your benchmark?

This mitigates Failure Mode 4: Prompts are collected from publicly available sources that are also likely to be in the datasets of SUT developers

Affirming Benchmarks:

Do you search the web for data included in your prompt set to find undisclosed collection from publicly available sources?

This mitigates Failure Mode 4: Prompts are collected from publicly available sources that are also likely to be in the datasets of SUT developers

Affirming Benchmarks:

Do you prohibit data vendors from re-licensing to you any data that they originally licensed from a third party?

This mitigates Failure Mode 5: Data vendor licenses private data that is available to other parties

Affirming Benchmarks:

Do you include contractual prohibitions against providing data that might be available to other parties?

This mitigates Failure Mode 5: Data vendor licenses private data that is available to other parties

Affirming Benchmarks:

Do you include contractual prohibitions against reselling licensed data to unrelated parties in your agreements with data vendors?

This mitigates Failure Mode 6: Data vendor provides same prompts to multiple organizations including benchmark authors and SUT developers

Affirming Benchmarks:

Do you disclose distributional limitations (e.g., that the benchmark does not provide information on distributional harms such as disparate impact) of the benchmark?

This mitigates Failure Mode 7: Singular prompts without a distributional tie lack a capacity for detecting distributional failings, harms, or impacts

Affirming Benchmarks:

Do you analyze the prompt length and/or the embedding space of prompts visually?

This mitigates Failure Mode 8: Prompt writers produce prompts with inadequate variability within the valid input space (e.g., a single prompt writer writes all the prompts)

Affirming Benchmarks:

Do you state sample counts in terms of root samples and templates (i.e., structured expressions of inputs subject to perturbation or instantiation)? Templates may be excluded entirely from the design.

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Do you state sample counts in terms of root samples and templates (i.e., structured expressions of inputs subject to perturbation or instantiation)? Templates may be excluded entirely from the design.

This mitigates Failure Mode 10: Prompt perturbation bulking (increasing the number of prompts by making small changes to root prompts)

Affirming Benchmarks:

Do you publicly state that the benchmark focuses on adversial testing or does the benchmark design sample beyond adversarial cases?

This mitigates Failure Mode 11: Prompts focus on adversarial users (e.g., users are attempting to circumvent a guard model)

Affirming Benchmarks:

Do you source prompts from SUTs deployed in a real world setting for the task under evaluation within the benchmark (i.e., a non-experimental setting)?

This mitigates Failure Mode 14: Benchmark does not capture the distribution or variability of the task in the real world

Affirming Benchmarks:

Do you avoid releasing a representative sample of prompts to SUT developers?

This mitigates Failure Mode 15: Prompts have known properties allowing for achieving an unrealistic (i.e., non-generalizing) performance. For example, prompts are of particular and known lengths.

Affirming Benchmarks:

Do you calibrate your prompt sample count to a statistical analysis indicating the number of samples required to identify the probability of rare events?

This mitigates Failure Mode 16: An inadequate number of prompts are produced to identify rare critical events (i.e., tail risks)

Affirming Benchmarks:

Do you prominently and publicly state the benchmark does not cover idioms within the languages supported by the benchmark?

This mitigates Failure Mode 18: No coverage for target language idiomatic expressions (including differences in functional expression, less common APIs, etc., within programming languages) beyond those known to the benchmark authors

Affirming Benchmarks:

Do you clearly state the cultural or contextual limitations of your benchmark, such as noting that definitions of violating content reflect a specific cultural perspective?

This mitigates Failure Mode 19: Cultural norms do not translate between cultural contexts (languages, geographies, etc.)

Affirming Benchmarks:

Do you validate all machine-translated prompts using a highly qualified speaker fluent in both source and target languages?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

(This mitigation may only be considered as applied when the prompts are never available to the SUT developer) Do you ensure that prompts are not shared with SUT developers during inferencing?

This mitigates Failure Mode 21: Prompts are sent to model vendors when inferencing or all prompts are publicly available

Affirming Benchmarks:

(This mitigation may only be considered as applied when the prompts are only processed under contract) Do you include contractual prohibitions both against logging test runs and for allowing human review of test prompts?

This mitigates Failure Mode 21: Prompts are sent to model vendors when inferencing or all prompts are publicly available

Affirming Benchmarks:

Do you perform a characteristic analysis comparing prompts collected from a production system to those in your test set, and do you publicly disclose any significant variations?

This mitigates Failure Mode 22: Distribution of SUT inputs within the real world are substantially different in distribution from those within the benchmark (e.g., SUT users ask different questions from those posed by the benchmark authors)

Affirming Benchmarks:

Do you provide sample prompts under a license that explicitly prohibits their use in model training?

This mitigates Failure Mode 23: SUT developer trains against sample prompt set

Affirming Benchmarks:

Do you refrain from making the evaluator or ground truth publicly available?

This mitigates Failure Mode 25: SUT developers place evaluator or other test ground truth within system chain

Affirming Benchmarks:

Do you validate all machine-translated prompts using a highly qualified speaker fluent in both source and target languages?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Do you measure and iterate on your methods until you achieve high interrater reliability for the human-labeled data used to train the evaluator, and ensure that any remaining uncertainty from disagreements is reflected in the confidence measures provided to users?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Do you statistically confirm performance for each new system under test (SUT) by evaluating the benchmark's accuracy on a representative sample of that SUT's outputs?

This mitigates Failure Mode 28: Evaluator tuned to SUT failing outputs and cannot generalize to new SUTs

Affirming Benchmarks:

Do you avoid making the evaluator or its answers publicly available?

This mitigates Failure Mode 29: SUT developers produce training data from evaluator

Affirming Benchmarks:

Do you calibrate the number of prompts used according to statistical analysis, or otherwise ensure that the sample count is sufficient to produce consistent and reliable measures?

This mitigates Failure Mode 34: Inadequate sample size for identifying performance

Affirming Benchmarks:

Do you prominently display uncertainty estimates alongside your benchmark results?

This mitigates Failure Mode 35: Failure to propagate uncertainty or confidence from lower level measures to higher level grades

Affirming Benchmarks:

Do you prominently display uncertainty estimates alongside your benchmark results?

This mitigates Failure Mode 36: Presentation without uncertainty or confidence of the scores

Affirming Benchmarks:

Do you prominently display disclaimers wherever the benchmark is presented?

This mitigates Failure Mode 37: User does not read disclaimers

Affirming Benchmarks:

Do you conduct user evaluations with your target user group and iterate on the benchmark design based on their feedback?

This mitigates Failure Mode 38: User does not understand visual representation of scores

Affirming Benchmarks:

Do you clearly and prominently display the scope of the benchmark, including guidance on what users should or should not rely on it for?

This mitigates Failure Mode 39: User misunderstands the scope of the benchmark

Affirming Benchmarks:

Do you have signed agreements with SUT developers that prohibit tuning systems using the provided sample set?

This mitigates Failure Mode 41: SUT developer tunes safety program to benchmark sample set

Affirming Benchmarks:

Do you intentionally avoid providing a representative sample set (i.e., a dataset illustrative of the characteristics of the prompts included within the test set)?

This mitigates Failure Mode 42: SUT developer trains SUT against sample set

Affirming Benchmarks:

Do you intentionally avoid providing the test set publicly or to SUT developers?

This mitigates Failure Mode 44: Test set leaks out to the general internet

Affirming Benchmarks:

Do you actively update your tests in response to changing user behaviors?

This mitigates Failure Mode 43: User behavior shifts through time

Affirming Benchmarks:

Do you actively monitor the internet to check whether any test set samples have appeared online?

This mitigates Failure Mode 44: Test set leaks out to the general internet

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 25: SUT developers place evaluator or other test ground truth within system chain

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 28: Evaluator tuned to SUT failing outputs and cannot generalize to new SUTs

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 29: SUT developers produce training data from evaluator

Affirming Benchmarks:

Do you identify and call out specific qualities of prompt subpopulations where SUT performance is systematically worse?

This mitigates Failure Mode 35: Failure to propagate uncertainty or confidence from lower level measures to higher level grades

Affirming Benchmarks:

Do you periodically re-test all systems under test (SUTs) to maintain up-to-date comparisons?

This mitigates Failure Mode 45: SUT developers update the SUT without changing the name or version of the SUT

Affirming Benchmarks:

Do you require that the system configuration (e.g., temperature, system prompts) used in benchmarking matches either the system’s default user-facing settings or those clearly recommended in vendor guidelines for the intended use case?

This mitigates Failure Mode 24: SUT is tested under conditions (e.g. temperature, iteration, context window settings) not matching deployment conditions or those conditions typically experienced by relying persons under default configurations

Affirming Benchmarks:

Do you conduct statistical analysis of evaluator (human, model, or algorithm) failures and prominently disclose any detected biases across all SUTs?

This mitigates Failure Mode 30: Certain SUTs produce outputs with higher evaluator errors than other SUTs

Affirming Benchmarks:

Do you iteratively improve your evaluator model and validate its performance until it reaches a predefined performance threshold?

This mitigates Failure Mode 31: Evaluator(s) perform poorly across all SUTs

Affirming Benchmarks:

Do you iteratively improve how disclaimers are displayed via UX research until users demonstrate understanding of the benchmark's limitations?

This mitigates Failure Mode 37: User does not read disclaimers

Affirming Benchmarks:

Do you intentionally avoid providing a representative sample set (i.e., a dataset illustrative of the characteristics of the prompts included within the test set)?

This mitigates Failure Mode 41: SUT developer tunes safety program to benchmark sample set

Affirming Benchmarks:

Do you use human evaluators for assessing system performance across all outputs evaluated by computational means when the outputs are evaluated as unacceptable?

This mitigates Failure Mode 32: Evaluator(s) have a high false reject rate (acceptable outputs rated as unacceptable)

Affirming Benchmarks:

Do you use human evaluators for assessing system performance across all outputs evaluated by computational means when the outputs are evaluated as acceptable?

This mitigates Failure Mode 33: Evaluator(s) have a high false accept rate (unacceptable outputs rated as acceptable)

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 32: Evaluator(s) have a high false reject rate (acceptable outputs rated as unacceptable)

Affirming Benchmarks:

Do you use 100% human evaluators for assessing system performance?

This mitigates Failure Mode 33: Evaluator(s) have a high false accept rate (unacceptable outputs rated as acceptable)

Affirming Benchmarks:

Do you produce data with distributional associations when designing your benchmark?

This mitigates Failure Mode 7: Singular prompts without a distributional tie lack a capacity for detecting distributional failings, harms, or impacts

Affirming Benchmarks:

Do you statistically validate the quality of translations and propagate uncertainty into the benchmark scores?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Do you prominently state that the benchmark does not fully cover the input space, and clarify its limitations in coverage?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you record the demographics of individuals generating prompts and make efforts to sample from underrepresented populations?

This mitigates Failure Mode 12: Prompt writers bias the sample to their own demographically-aligned word use, topics of interest, or other dimensions that tend to not explore the entirety of supported input space for the benchmark's supported use case

Affirming Benchmarks:

Do you localize the presentation of benchmark results in consultation with individuals who have deep understanding of how such information is interpreted within specific demographic groups?

This mitigates Failure Mode 40: Different demographic groups (cultural, professional, educational, etc.) viewing the benchmark have different interpretations of the information conveyed

Affirming Benchmarks:

Do you restrict or avoid evaluation on demand to preserve benchmark integrity?

This mitigates Failure Mode 46: SUT developers can run the benchmark an unlimited number of times

Affirming Benchmarks:

Do you reduce the scope of the benchmark to ensure comprehensive coverage of the input space is achievable?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you source prompts from multiple populations with clearly distinct demographic characteristics among prompt writers?

This mitigates Failure Mode 8: Prompt writers produce prompts with inadequate variability within the valid input space (e.g., a single prompt writer writes all the prompts)

Affirming Benchmarks:

Do you generate a sufficient number of human-written prompts to enable statistical comparison with LLM-generated prompts, and do you conduct a comparative study?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Do you source prompts from a range of different geographic regions to increase representativeness?

This mitigates Failure Mode 12: Prompt writers bias the sample to their own demographically-aligned word use, topics of interest, or other dimensions that tend to not explore the entirety of supported input space for the benchmark's supported use case

Affirming Benchmarks:

Do you verify with engineering staff of SUT developers that prompts will not be recorded? You may only consider this mitigation as applied when all relevant SUT engineering staffs affirm this.

This mitigates Failure Mode 21: Prompts are sent to model vendors when inferencing or all prompts are publicly available

Affirming Benchmarks:

Do you employ a backup evaluator with human validation of sampled outputs to ensure evaluation reliability?

This mitigates Failure Mode 25: SUT developers place evaluator or other test ground truth within system chain

Affirming Benchmarks:

Do you periodically introduce new test set prompts and monitor for statistical differences in performance between new and old items?

This mitigates Failure Mode 44: Test set leaks out to the general internet

Affirming Benchmarks:

Do you perform design studies with potential users to understand presentation requirements for benchmark outputs?

This mitigates Failure Mode 38: User does not understand visual representation of scores

Affirming Benchmarks:

Does your license prohibit tuning to the sample set?

This mitigates Failure Mode 41: SUT developer tunes safety program to benchmark sample set

Affirming Benchmarks:

Do you have signed agreements with SUT developers that explicitly prohibit training with the sample set?

This mitigates Failure Mode 42: SUT developer trains SUT against sample set

Affirming Benchmarks:

Does your license prohibit tuning to the sample set?

This mitigates Failure Mode 42: SUT developer trains SUT against sample set

Affirming Benchmarks:

Do you analyze prompt properties—such as length and embedding space distribution—using visual or statistical methods?

This mitigates Failure Mode 15: Prompts have known properties allowing for achieving an unrealistic (i.e., non-generalizing) performance. For example, prompts are of particular and known lengths.

Affirming Benchmarks:

Do you source prompts from multiple individuals representing distinct populations to ensure diversity and reduce bias?

This mitigates Failure Mode 15: Prompts have known properties allowing for achieving an unrealistic (i.e., non-generalizing) performance. For example, prompts are of particular and known lengths.

Affirming Benchmarks:

For all languages benchmarked, do you validate evaluator versus native speaker response grading to ensure the error rate is consistent with the representations made to users?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Do you direct the benchmark to a professional audience, providing layers of supporting information suitable for expert review?

This mitigates Failure Mode 40: Different demographic groups (cultural, professional, educational, etc.) viewing the benchmark have different interpretations of the information conveyed

Affirming Benchmarks:

Do you publicly disclaim the benchmark as a tool intended for consumer use?

This mitigates Failure Mode 40: Different demographic groups (cultural, professional, educational, etc.) viewing the benchmark have different interpretations of the information conveyed

Affirming Benchmarks:

Do you analyze prompt length and embedding space distribution visually to understand prompt variability?

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Do you analyze prompt length and embedding space distribution visually to understand prompt variability?

This mitigates Failure Mode 10: Prompt perturbation bulking (increasing the number of prompts by making small changes to root prompts)

Affirming Benchmarks:

Do you source prompts from multiple populations with distinct demographic attributes of the prompt writers?

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Do you source prompts from multiple populations with distinct demographic attributes of the prompt writers?

This mitigates Failure Mode 10: Prompt perturbation bulking (increasing the number of prompts by making small changes to root prompts)

Affirming Benchmarks:

Do you limit promotion of the benchmark to individuals with a sophisticated understanding of the task being benchmarked and its measurement implications?

This mitigates Failure Mode 40: Different demographic groups (cultural, professional, educational, etc.) viewing the benchmark have different interpretations of the information conveyed

Affirming Benchmarks:

Are all prompts authored by the benchmark creators themselves, without using data vendors, LLMs, or crowd workers whose identities are unknown to the authors?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Does the benchmark avoid presenting results for adversarial prompts or scenarios where adversarial users are not applicable?

This mitigates Failure Mode 11: Prompts focus on adversarial users (e.g., users are attempting to circumvent a guard model)

Affirming Benchmarks:

Do you include canary data in the prompt set that would reveal if the benchmark was used in model training, enabling detection of training contamination?

This mitigates Failure Mode 23: SUT developer trains against sample prompt set

Affirming Benchmarks:

Do you avoid using data vendors and machine translation in prompt generation, ensuring all prompts originate in the language of the test?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Is the evaluator strictly algorithmic (i.e., applying a list of correct answers) with no legitimate reason to be embedded in the system-under-test (SUT) chain?

This mitigates Failure Mode 25: SUT developers place evaluator or other test ground truth within system chain

Affirming Benchmarks:

Can you confirm that no data vendors were used and that the evaluation data was not machine-translated, based on direct observation of practices?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Does the evaluator apply an algorithm to compare model outputs against a list of correct answers to assess correctness or compliance?

This mitigates Failure Mode 28: Evaluator tuned to SUT failing outputs and cannot generalize to new SUTs

Affirming Benchmarks:

Does the evaluator apply an algorithm to compare model outputs against a list of correct answers to assess correctness or compliance?

This mitigates Failure Mode 29: SUT developers produce training data from evaluator

Affirming Benchmarks:

Is the system-under-test required to produce outputs that conform to the evaluator’s expected format (e.g., true/false, multiple choice)?

This mitigates Failure Mode 30: Certain SUTs produce outputs with higher evaluator errors than other SUTs

Affirming Benchmarks:

Is the system-under-test required to produce outputs that conform to the evaluator’s expected format (e.g., true/false, multiple choice)?

This mitigates Failure Mode 31: Evaluator(s) perform poorly across all SUTs

Affirming Benchmarks:

Is the system-under-test required to produce outputs that conform to the evaluator’s expected format (e.g., true/false, multiple choice)?

This mitigates Failure Mode 32: Evaluator(s) have a high false reject rate (acceptable outputs rated as unacceptable)

Affirming Benchmarks:

Is the system-under-test required to produce outputs that conform to the evaluator’s expected format (e.g., true/false, multiple choice)?

This mitigates Failure Mode 33: Evaluator(s) have a high false accept rate (unacceptable outputs rated as acceptable)

Affirming Benchmarks:

Do you ensure that all prompts are produced solely through original human effort without sourcing data from external sources?

This mitigates Failure Mode 5: Data vendor licenses private data that is available to other parties

Affirming Benchmarks:

Do you supervise prompt production by requiring that any external party generates only novel, previously unwritten, and undigitized prompts?

This mitigates Failure Mode 6: Data vendor provides same prompts to multiple organizations including benchmark authors and SUT developers

Affirming Benchmarks:

Do you describe how the tested capability or concept is translated into the benchmarked task?

This mitigates Failure Mode 47: The benchmark does not measure a property of the SUT linked to the user task

Affirming Benchmarks:

Are domain experts involved in the creation of the benchmark?

This mitigates Failure Mode 47: The benchmark does not measure a property of the SUT linked to the user task

Affirming Benchmarks:

Do you integrate relevant domain literature into the benchmark?

This mitigates Failure Mode 47: The benchmark does not measure a property of the SUT linked to the user task

Affirming Benchmarks:

Do you describe the use cases or the user personas of those who will rely on the benchmark?

This mitigates Failure Mode 1: The information provided by the benchmark does not match with the information the benchmark user believes is provided

Affirming Benchmarks:

Are metric floors and ceilings included as part of the benchmark methodology?

This mitigates Failure Mode 48: Users cannot map the scores to a mental model of likely SUT behavior in the real world

Affirming Benchmarks:

Is a human performance level incorporated into the benchmark?

This mitigates Failure Mode 48: Users cannot map the scores to a mental model of likely SUT behavior in the real world

Affirming Benchmarks:

Is a performance level for random decision/output also provided in the benchmark?

This mitigates Failure Mode 48: Users cannot map the scores to a mental model of likely SUT behavior in the real world

Affirming Benchmarks:

Do you add a globally unique identifier or apply encryption to evaluation instances?

This mitigates Failure Mode 49: SUT developer trains against evaluation set prior to benchmark release

Affirming Benchmarks:

Do you add a globally unique identifier or apply encryption to evaluation instances?

This mitigates Failure Mode 50: SUT developer trains against evaluation set after benchmark release

Affirming Benchmarks:

Do you refrain from providing the evaluation set to prevent potential misuse?

This mitigates Failure Mode 49: SUT developer trains against evaluation set prior to benchmark release

Affirming Benchmarks:

Are there signed agreements with SUT developers that prohibit using the sample set for training?

This mitigates Failure Mode 49: SUT developer trains against evaluation set prior to benchmark release

Affirming Benchmarks:

Does the license explicitly prohibit using the sample set for training, even if the sample set is available without a signed agreement?

This mitigates Failure Mode 49: SUT developer trains against evaluation set prior to benchmark release

Affirming Benchmarks:

Do you refrain from providing the evaluation set?

This mitigates Failure Mode 50: SUT developer trains against evaluation set after benchmark release

Affirming Benchmarks:

Are there signed agreements with SUT developers that specifically prohibit training with the evaluation set?

This mitigates Failure Mode 50: SUT developer trains against evaluation set after benchmark release

Affirming Benchmarks:

Does the license explicitly prohibit training with the evaluation set, even when it might be available without a signed agreement?

This mitigates Failure Mode 50: SUT developer trains against evaluation set after benchmark release

Affirming Benchmarks:

Do you include canary data in the prompt set such that its inclusion in training would enable testing whether the model has been trained on the benchmark?

This mitigates Failure Mode 49: SUT developer trains against evaluation set prior to benchmark release

Affirming Benchmarks:

Do you include canary data in the prompt set such that its inclusion in training would enable testing whether the model has been trained on the benchmark?

This mitigates Failure Mode 50: SUT developer trains against evaluation set after benchmark release

Affirming Benchmarks:

Are the release requirements such as the practices and norms required to be followed by the SUT developer specified publicly?

This mitigates Failure Mode 51: SUT developers are not bound to adhere to benchmark integrity requirements

Affirming Benchmarks:

Are SUT developers held legally and financially accountable for harms that might arise from violating benchmark release requirements (i.e., if training practices producing a benchmark score not representative of SUT performance in the real world)?

This mitigates Failure Mode 51: SUT developers are not bound to adhere to benchmark integrity requirements

Affirming Benchmarks:

Is there an ongoing, maintained feedback channel available for benchmark users?

This mitigates Failure Mode 39: User misunderstands the scope of the benchmark

Affirming Benchmarks:

Is a contact person listed for the benchmark, and is that person responsive to inquiries?

This mitigates Failure Mode 39: User misunderstands the scope of the benchmark

Affirming Benchmarks:

Has the accompanying paper been accepted at a peer-reviewed venue, or is there a preprint version that is highly cited?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Is the benchmark design process documented?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Are the test tasks and the rationale behind them documented?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Are any assumptions about normative properties (e.g., how the benchmark should be interpreted within specific cultures or contexts) clearly documented?

This mitigates Failure Mode 40: Different demographic groups (cultural, professional, educational, etc.) viewing the benchmark have different interpretations of the information conveyed

Affirming Benchmarks:

Is the design process for the test environment or prompts (e.g., how prompts are written and by whom) documented?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Are the data sources and the data collection process explained?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Are any data preprocessing steps described or found to be non-relevant to benchmark users?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Are any data annotation steps described or found to be non-relevant to benchmark users?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Is the evaluation metric used by the benchmark clearly documented?

This mitigates Failure Mode 52: Benchmark production failed to account for an idiosyncratic failure mode

Affirming Benchmarks:

Is the representativeness of the data explained (e.g., are the demographics of crowd workers collected and reported)?

This mitigates Failure Mode 53: Linkage between the evaluation prompts and the information the prompts are meant to supply via the benchmark is not well understood by the benchmark user

Affirming Benchmarks:

Does the benchmark avoid processing human languages in its inputs and outputs?

This mitigates Failure Mode 19: Cultural norms do not translate between cultural contexts (languages, geographies, etc.)

Affirming Benchmarks:

Is there evidence of high agreement between human annotation outputs and the evaluation model?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Is the benchmark periodically reviewed to ensure the SUT task aligns with current user expectations, including updating prompts, sourcing new ones for better coverage, and releasing versioned updates?

This mitigates Failure Mode 54: New requirements emerge that would reasonably be interpreted as being covered in the task definition

Affirming Benchmarks:

Do you measure and iterate on methods until high interrater reliability is achieved?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Is the evaluator observed to reliably achieve a low false reject rate (e.g., compliant labeled as non-compliance)?

This mitigates Failure Mode 32: Evaluator(s) have a high false reject rate (acceptable outputs rated as unacceptable)

Affirming Benchmarks:

Is the evaluator observed to reliably achieve a low false accept rate (e.g., non-compliant labeled as compliance)?

This mitigates Failure Mode 33: Evaluator(s) have a high false accept rate (unacceptable outputs rated as acceptable)

Affirming Benchmarks:

Do you produce entirely new prompts with the aid of humans?

This mitigates Failure Mode 4: Prompts are collected from publicly available sources that are also likely to be in the datasets of SUT developers

Affirming Benchmarks:

Are all prompts entirely new and produced in-house by the benchmark developers?

This mitigates Failure Mode 6: Data vendor provides same prompts to multiple organizations including benchmark authors and SUT developers

Affirming Benchmarks:

Do you define the information foraging task of the benchmark-relying user in a way that allows for sampling over the input space rather than requiring comprehensive coverage?

This mitigates Failure Mode 8: Prompt writers produce prompts with inadequate variability within the valid input space (e.g., a single prompt writer writes all the prompts)

Affirming Benchmarks:

Do you source additional prompts through peer review and/or submissions from the broader community?

This mitigates Failure Mode 14: Benchmark does not capture the distribution or variability of the task in the real world

Affirming Benchmarks:

Do you source prompts from SUTs deployed to real users under realistic laboratory conditions (i.e., the user is directed to do a real task)?

This mitigates Failure Mode 14: Benchmark does not capture the distribution or variability of the task in the real world

Affirming Benchmarks:

Do you avoid releasing the test set to SUT developers?

This mitigates Failure Mode 15: Prompts have known properties allowing for achieving an unrealistic (i.e., non-generalizing) performance. For example, prompts are of particular and known lengths.

Affirming Benchmarks:

Is it established in the task definition that rare events (i.e., tail risks) are not of interest to the benchmark-relying person?

This mitigates Failure Mode 16: An inadequate number of prompts are produced to identify rare critical events (i.e., tail risks)

Affirming Benchmarks:

Does the benchmark avoid processing human languages in its inputs or outputs?

This mitigates Failure Mode 18: No coverage for target language idiomatic expressions (including differences in functional expression, less common APIs, etc., within programming languages) beyond those known to the benchmark authors

Affirming Benchmarks:

Does the benchmark avoid processing human languages in its inputs or outputs?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Do you require SUT developers to disclose their training practices when attempting to maximize their scores, including any training or engineering done to specifically target the sample set? And do you require them to disclaim results in such cases?

This mitigates Failure Mode 23: SUT developer trains against sample prompt set

Affirming Benchmarks:

Do you avoid providing a sample set that would allow for tuning hyperparameters on the test set and do you avoid publishing the evaluation set?

This mitigates Failure Mode 24: SUT is tested under conditions (e.g. temperature, iteration, context window settings) not matching deployment conditions or those conditions typically experienced by relying persons under default configurations

Affirming Benchmarks:

Do you allow SUT developers a very limited number of evaluations?

This mitigates Failure Mode 24: SUT is tested under conditions (e.g. temperature, iteration, context window settings) not matching deployment conditions or those conditions typically experienced by relying persons under default configurations

Affirming Benchmarks:

Does the benchmark avoid testing concepts related to language (i.e., inputs and outputs are not textual)?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Does the benchmark use singular answers that are checked against a list?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Does the benchmark use singular answers that are checked against a list?

This mitigates Failure Mode 28: Evaluator tuned to SUT failing outputs and cannot generalize to new SUTs

Affirming Benchmarks:

Does the benchmark use singular answers (i.e., bit-exact outputs) that are checked via a lookup table?

This mitigates Failure Mode 30: Certain SUTs produce outputs with higher evaluator errors than other SUTs

Affirming Benchmarks:

Does the benchmark use singular answers (i.e., bit-exact outputs) that are checked via a lookup table?

This mitigates Failure Mode 31: Evaluator(s) perform poorly across all SUTs

Affirming Benchmarks:

Does the benchmark use singular answers (i.e., bit-exact outputs) that are checked via a lookup table?

This mitigates Failure Mode 32: Evaluator(s) have a high false reject rate (acceptable outputs rated as unacceptable)

Affirming Benchmarks:

Does the benchmark use singular answers (i.e., bit-exact outputs) that are checked via a lookup table?

This mitigates Failure Mode 33: Evaluator(s) have a high false accept rate (unacceptable outputs rated as acceptable)

Affirming Benchmarks:

Is the property of interest appropriately measurable without aggregating across different subpopulations?

This mitigates Failure Mode 35: Failure to propagate uncertainty or confidence from lower level measures to higher level grades

Affirming Benchmarks:

Do you limit the presentation and audience to sophisticated users who can be assumed to understand visual representations?

This mitigates Failure Mode 38: User does not understand visual representation of scores

Affirming Benchmarks:

Do you only test SUTs with well-known version identifiers?

This mitigates Failure Mode 45: SUT developers update the SUT without changing the name or version of the SUT

Affirming Benchmarks:

Do you list and make available a complete definition of what is and isn’t covered by the benchmark?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you construct the task definition such that LLMs are of limited utility in producing the prompts?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Do you involve domain experts in producing the prompts to ensure all known forms of variation are covered by the inputs?

This mitigates Failure Mode 8: Prompt writers produce prompts with inadequate variability within the valid input space (e.g., a single prompt writer writes all the prompts)

Affirming Benchmarks:

Is the task defined in a way that does not examine adversarial performance, and are adversarial prompts excluded?

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Does the task definition aim to avoid processing and expression of idioms?

This mitigates Failure Mode 18: No coverage for target language idiomatic expressions (including differences in functional expression, less common APIs, etc., within programming languages) beyond those known to the benchmark authors

Affirming Benchmarks:

Do you publish the benchmark at an organization that does not develop SUTs (i.e., is it a third-party benchmark)?

This mitigates Failure Mode 55: A SUT developer has disparate access to information about the benchmark **after** its release (i.e., information not provided to other SUT developers)

Affirming Benchmarks:

Do you require all benchmark authors and prompt writers to maintain confidentiality of non-public information regarding the benchmark?

This mitigates Failure Mode 55: A SUT developer has disparate access to information about the benchmark **after** its release (i.e., information not provided to other SUT developers)

Affirming Benchmarks:

Do you prohibit benchmark authors from optimizing SUTs for the benchmark?

This mitigates Failure Mode 55: A SUT developer has disparate access to information about the benchmark **after** its release (i.e., information not provided to other SUT developers)

Affirming Benchmarks:

Is your organization a SUT developer and do you maintain an information barrier protocol (e.g., Chinese wall) between benchmark authors/prompt writers and SUT developers within the same organization?

This mitigates Failure Mode 55: A SUT developer has disparate access to information about the benchmark **after** its release (i.e., information not provided to other SUT developers)

Affirming Benchmarks:

Do you involve domain experts in defining the task and prompts?

This mitigates Failure Mode 56: The benchmark authors do not know how to formulate the problem as prompts that are illustrative to the user relying on the benchmark

Affirming Benchmarks:

Does at least one benchmark author have formal training in statistical methods?

This mitigates Failure Mode 57: Benchmark authors do not know how to propagate statistical uncertainty into a user presentation

Affirming Benchmarks:

Do you provide comparisons to well-known benchmarks that indicate what is the same and what is different?

This mitigates Failure Mode 58: Understanding the benchmark requires more resources (e.g., study, expertise, exploration) than the relying user has time to expend

Affirming Benchmarks:

Do you publish a FAQ along with the benchmark?

This mitigates Failure Mode 58: Understanding the benchmark requires more resources (e.g., study, expertise, exploration) than the relying user has time to expend

Affirming Benchmarks:

Do you provide simple interpretative statements for scores in the user interface?

This mitigates Failure Mode 58: Understanding the benchmark requires more resources (e.g., study, expertise, exploration) than the relying user has time to expend

Affirming Benchmarks:

Have you simplified the task to reduce the mental effort required to understand it?

This mitigates Failure Mode 58: Understanding the benchmark requires more resources (e.g., study, expertise, exploration) than the relying user has time to expend

Affirming Benchmarks:

Do you include example prompts for the task?

This mitigates Failure Mode 58: Understanding the benchmark requires more resources (e.g., study, expertise, exploration) than the relying user has time to expend

Affirming Benchmarks:

Are all prompts sourced from real users under normal circumstances?

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Are the source prompts taken from systems under test (SUTs) deployed in real-world, non-experimental settings for the task under evaluation?

This mitigates Failure Mode 18: No coverage for target language idiomatic expressions (including differences in functional expression, less common APIs, etc., within programming languages) beyond those known to the benchmark authors

Affirming Benchmarks:

Do you source prompts from experimental settings that enable the collection of idioms?

This mitigates Failure Mode 18: No coverage for target language idiomatic expressions (including differences in functional expression, less common APIs, etc., within programming languages) beyond those known to the benchmark authors

Affirming Benchmarks:

Is the benchmark data sourced entirely from a single language, and do you make no claims regarding its applicability to other languages?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Did you use an off-the-shelf evaluator model without tuning, while ensuring the evaluation program demonstrates adequate reliability for the task?

This mitigates Failure Mode 27: Low interrater reliability of ground truth data used to tune the evaluator

Affirming Benchmarks:

Is your data collected from publicly available sources with no incentive or directive to adversarially inflate the number of prompts?

This mitigates Failure Mode 9: Adversarial prompt bulking (increasing the number of prompts by multiplying them by the number of tactics)

Affirming Benchmarks:

Is your data collected from publicly available sources with no incentive or directive to adversarially inflate the number of prompts?

This mitigates Failure Mode 10: Prompt perturbation bulking (increasing the number of prompts by making small changes to root prompts)

Affirming Benchmarks:

Is all the data of naturalistic origin (e.g., collected from public forums)?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Do you provide a task taxonomy that shows how the input/output space maps to hazards, failures, or compliant/non-compliant prompts and outputs?

This mitigates Failure Mode 1: The information provided by the benchmark does not match with the information the benchmark user believes is provided

Affirming Benchmarks:

Have you split the benchmark into smaller, focused subtasks and presented scores for those subtasks separately?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you sample prompts from the input space according to their real-world likelihood of occurrence (e.g., based on analysis of deployed systems)?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you visualize and share what parts of the input space are covered versus missing?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you report metrics indicating coverage of the task space?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Have you red-teamed the evaluation to identify undercovered input space regions and generated additional data for those areas?

This mitigates Failure Mode 2: The task is defined too broadly to achieve any reasonable degree of coverage over the use case

Affirming Benchmarks:

Do you audit test set prompts for issues such as signs of undisclosed use of LLMs in their generation?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Were domain experts involved in benchmark construction with the goal of capturing the full variability of the task?

This mitigates Failure Mode 14: Benchmark does not capture the distribution or variability of the task in the real world

Affirming Benchmarks:

Do you tag prompts with their cultural context and explicitly disclaim contexts that are not covered?

This mitigates Failure Mode 19: Cultural norms do not translate between cultural contexts (languages, geographies, etc.)

Affirming Benchmarks:

Do you check whether back-translating from the target language to the source language yields expressions consistent with the original language?

This mitigates Failure Mode 20: Producing prompts in a language from prompts translated from another language introduces errors

Affirming Benchmarks:

Do you check whether back-translating from the target language to the source language yields expressions consistent with the original language?

This mitigates Failure Mode 26: Evaluator (humans labeling the final outputs used in benchmarking or an LLM-as-a-judge) tuned on translated outputs with substantial errors

Affirming Benchmarks:

Have you implemented interactive tutorials or help sections that explain the visual representations used in the benchmarks?

This mitigates Failure Mode 38: User does not understand visual representation of scores

Affirming Benchmarks:

Do you clearly, publicly, and prominently state the user information foraging task associated with the evaluation, even if only for sophisticated users such as LLM engineers?

This mitigates Failure Mode 1: The information provided by the benchmark does not match with the information the benchmark user believes is provided

Affirming Benchmarks:

Was the benchmark produced before the widespread availability of LLMs capable of generating its prompts?

This mitigates Failure Mode 3: Input prompt writers produce prompts with LLMs

Affirming Benchmarks:

Does the benchmark provide guidance or references on how to display or interpret statistics that communicate benchmark uncertainty?

This mitigates Failure Mode 36: Presentation without uncertainty or confidence of the scores

Affirming Benchmarks:

Are the prompt outputs multi-step, inherently high dimensional, or possess some other property making the benchmarked task difficult to game on the basis of a non-generalizing input feature (e.g., prompt lengths)?

This mitigates Failure Mode 15: Prompts have known properties allowing for achieving an unrealistic (i.e., non-generalizing) performance. For example, prompts are of particular and known lengths.

Affirming Benchmarks:

Did you use an off-the-shelf evaluator model without tuning, but with an evaluation program that finds its reliability to be adequate for the evaluation task?

This mitigates Failure Mode 31: Evaluator(s) perform poorly across all SUTs

Affirming Benchmarks: