Purpose and fit

Use this for an initial check or to compare a proposed workflow change with the current version. A small set is useful for finding concrete defects and missing cases. Treat it as a starting point for evaluation.

Before you start

Start with real cases you already hold and are authorised to use, after suitable redaction or de-identification. Tell the AI the fields, task types and sample coverage actually available. Use fewer cases if that is what exists and state the limitation. You do not need to create a synthetic dataset; invented examples are optional, clearly labelled supplements and never observations.

Use only information you are authorised to share with the chosen AI tool. Adjust the prompt to the fields and context you actually have; a specially created dataset is not required.

Required inputs

  • A bounded workflow and the proposed change
  • Existing authorised cases or case descriptions, with suitable redaction/de-identification and clear provenance; optional synthetic examples must be labelled separately
  • A task-specific acceptance rubric and known failure types
  • A controlled way to record outputs, versions, settings and reviews

Practical workflow

  1. Describe the tasks and conditions the evaluation is meant to cover. Identify ordinary cases, difficult cases and cases outside the allowed scope.
  2. Select a manageable starter set from the authorised cases available. Aim to inspect routine, difficult and stop/ask situations where those cases exist, recording missing coverage. Twelve cases is only an illustration, not a required or validated sample size; do not manufacture cases to meet a quota.
  3. Have a domain reviewer define expected behaviour and the evidence for it. Keep answer keys and evaluation notes separate from the system’s task inputs. Preserve normal approved context that the real workflow would receive.
  4. Run the versions being compared under recorded conditions. Where output variation matters, choose and record a repeat-run plan before interpreting results. Keep failures and incomplete attempts in the log.
  5. Review outputs using the same rubric. Report counts by case type, severity and run. Investigate failures, then test changes on fresh cases as well as the original set.

Expected output

Expect an evaluation-plan draft, case-selection worksheet and blank results log unless real results are supplied. Designing a set is not running an evaluation, and a small selected set does not establish reliability.

An evaluation worksheet and results log:

Case ID | case type | provenance | approved input | expected behaviour/source | critical failure | workflow/version/settings | run ID | observed output | result/reason | reviewer | unresolved issue

A short decision summary states what was tested, what failed, what changed and what remains untested. Keep routine and deliberately difficult cases visible separately; the test mix may not reflect production frequency.

Human checks

A domain reviewer verifies answer keys and difficult judgments. Check for accidental answer leakage, missing cases, selective exclusions and changes in the scoring standard. Track cases used for tuning so they are not presented later as untouched validation evidence. Verify that any tested stop or approval boundary is also enforced by the real workflow.

Limitations

A small, selected set cannot estimate rare harms or establish production reliability. Synthetic examples may miss real complexity. Passing results do not justify a scope expansion without the relevant permissions, controls and further evidence.

Starter prompt

Help design an initial evaluation set for [workflow] and the proposed change [change]. The task types are [types], boundaries are [rules], and acceptance rubric is [rubric].

Use these available authorised, sanitised case descriptions: [insert cases, fields and provenance, or state that none are available]. Suggest how to cover ordinary work, ambiguity, missing/conflicting evidence, tool failure and stop/approval situations using these cases. Record absent categories as gaps. If I explicitly request illustrative invented examples, label each “synthetic” and keep it separate from real cases and results. For each case specify expected behaviour, source evidence and critical failures. Keep answer keys separate from task inputs. Suggest a blank results log and state what this small set cannot establish. Do not claim statistically valid sampling, executed tests or validated reliability.

First map the available case descriptions to ordinary, difficult and stop/ask categories. Keep missing categories as coverage gaps rather than inventing records. For each selected case specify expected behaviour, its evidence and the reviewer check. Leave run results blank unless supplied; separate proposed cases, actual cases and actual observations.

Before drafting, ask a focused question if a missing detail materially changes the task or acceptance criteria. Otherwise produce a clearly limited draft and list the gaps. Label any non-critical assumptions; never invent facts, figures, sources, approvals or completed actions.

Data and AI output disclaimer

Use relevant data that exists or that you provide, with accuracy and completeness sufficient for the task. Different AI models and versions may produce different results, and responses may vary between runs. You are responsible for checking and validating the output against your own requirements, source information and acceptance criteria before relying on it.