Publish a methods card readers can check

What should a reader be able to learn from a public methods card before relying on the headline result?

A reader should be able to tell what was tested, under which conditions, against what reference answer and for which uses the result is relevant. Put the most important limitation beside the result and give the card a version and date.

Describe the experiment, not just the product

Start with a narrow question and the decision it informs. Name the tested workflow, model or product version if known, tools enabled, run date and source of the test materials. Explain how examples were selected, how duplicates were handled and what was held back. A card for one experiment describes that experiment, even when the model itself has a broader range of uses.

Describe the reference answers. Who wrote them? Was each answer checked, and how were disagreements resolved? Say which materials readers may inspect and which are restricted. A public summary can give counts, selection criteria and an authorized synthetic example when the underlying material cannot be shared.

Give the result a denominator and a boundary

Show the overall numerator and denominator along with the outcome for the cases that matter most. Explain whether a system that declined to answer earned credit and how missing or unscorable examples were handled. Describe at least one failure in enough detail that a reader can recognize it.

State whose documents, languages or conditions were absent. If you did not test handwritten scans, a result on clean text offers little evidence about those scans. Note inaccessible settings, changes that would require a new test and where a dated correction can be found.

Make the publication traceable

Assign a card identifier and version. Identify the reviewer or review role accurately. Link only to materials the publisher is entitled to release and check the links from a reader’s perspective. Keep revisions dated. A corrected denominator should identify the previous value and explain the correction. The original model-card research is a useful precedent for describing intended use and evaluation conditions.

Primary sources

• Mitchell and colleagues · Model Cards for Model Reporting
• NIST AI Risk Management Framework Playbook · Measure

Related work at Agentic Attorney

This methods-card guide participates in the shared Agentic Attorney resource collection. Agentic Attorney describes its consulting and implementation work for legal teams. Explore Agentic Attorney’s approach to AI implementation.