Replayability and reproducibility are not the same thing

Replayability describes the ability to execute the same defined process or verification question again.

Reproducibility describes the ability to recover the same relevant result under a declared basis.

In AI systems, these concepts must be separated because a replayed probabilistic model may not produce identical free-form text even when the surrounding evidence and verification conditions are unchanged.

Replayability: can the process be run again?

A workflow is replayable when the information required to execute it again has been preserved.

This usually includes the input references, relevant state, configuration, model or service identities, transformation order and verification rules.

Replayability is therefore primarily about recoverability of process.

Reproducibility: can the relevant result be recovered?

Reproducibility focuses on outcome rather than only execution.

The relevant outcome depends on the verification question. In one system it may be a canonical accept or reject result. In another it may be a structured evidence state, integrity result or policy evaluation.

The key is to define what must remain invariant before claiming reproducibility.

Repeatability is narrower

Repeatability is often used for repeated execution under the same environment and conditions.

A test may be repeatable on the same machine, runtime and configuration while not yet being reproducible across an independent environment.

This distinction matters when evidence needs to survive beyond the original system or operator.

Probabilistic generation complicates the terminology

Generative AI systems may vary output even when prompts and source context appear unchanged.

Sampling behaviour, model implementation, runtime state and provider-side changes can all influence free-form generation.

That makes byte-identical output a poor default definition of reproducibility for many AI-assisted workflows.

Define the invariant first

Reproducibility claims become useful only when the invariant is explicit.

For an evidence workflow, the invariant may be the source set, provenance graph, structured state or canonical verification result rather than the exact wording of a model response.

Once the invariant is declared, replay can be evaluated against the correct property.

A replay basis must be preserved

A replayable system needs enough recorded information to reconstruct the execution or verification context.

  • source and artefact identities
  • versions or immutable references
  • model, agent and service identities
  • relevant configuration
  • policy or rule versions
  • ordered transformations
  • material system state
  • verification boundary

Replay without provenance can be misleading

A workflow may be technically rerunnable while still lacking evidence that the replay used the same source basis as the original execution.

Provenance connects replayed inputs to the historical artefacts that mattered in the original process.

Without that link, a rerun may look equivalent while operating on changed or substituted evidence.

Replay without state can produce false comparisons

Model version, retrieval state, policy configuration and runtime parameters may all influence the result.

If those factors are not preserved, a later replay can answer a different question while appearing to repeat the original one.

State is therefore part of the replay basis whenever it materially affects the outcome.

Canonical verification results make reproducibility practical

A structured canonical result gives replay a stable comparison target.

Rather than comparing free-form explanations, the system can compare the verification identifier, state basis, rule outcomes, reasons and overall canonical result.

This creates a clearer distinction between variable presentation and invariant verification semantics.

Replaying an AI workflow is not the same as replaying a verifier

A complete AI workflow may contain probabilistic generation, external APIs, human judgment and changing data sources.

A verifier can still be replayable and reproducible even when the larger workflow is not fully deterministic.

This is one reason verification should be treated as its own bounded system rather than as an informal property of the entire AI application.

A useful example

Consider an AI-assisted review where a model analyses a fixed evidence package and a deterministic verifier checks whether required sources, identities and review steps are present.

A later replay may produce slightly different model wording. The workflow is still useful to reproduce if the same evidence package, state and review conditions can be reconstructed.

The canonical verification result should remain the same if the declared verification inputs and rules are unchanged.

Independent reproduction is stronger than local replay

A process replayed successfully by the original system provides useful evidence.

An independent implementation that evaluates the same canonical verification basis provides stronger evidence because it reduces dependence on hidden local behaviour.

This is especially valuable for interoperability, audit and cross-system validation.

What should be reproducible?

The answer depends on the workflow, but several targets are often more useful than byte-identical model output.

  • evidence identity
  • provenance relationships
  • declared state basis
  • verification rule set
  • structured rule outcomes
  • canonical verification result
  • integrity evidence
  • declared boundary and scope

Start with one replay question

A practical test is to take one historical AI-assisted decision and ask whether the relevant verification question can be evaluated again.

Can the original evidence basis be recovered? Can the relevant state be reconstructed? Are the same rules available? Can the canonical result be compared?

If not, the gap reveals which evidence or state must be preserved to make future decisions replayable and reproducible.