Skip to content
Merulan

Run and compare a study

Run a pinned Experiment, inspect each subject's evidence, and compare candidates without deploying the selected result.

Use this guide when you test a change with Labs & Modeling. As a builder, pin a study, run its subjects, and compare the evidence before proposing live work.

Before you start

The study needs an objective version, saved Cases, exact subject versions, a method, and an evaluation bundle. Confirm each saved reference resolves in the active Workspace. Choose the test mode from Experiments and evaluations before starting. The mode determines which evidence the test needs. Prepare a Sandbox for separate scenario data.

Pin the study

In Laboratory, compose the Experiment with the selected references. The create contract records its revision. Open it at /labs/experiments. Check the objective, Case set, method, test mode, and composition digest. Keep that exact revision for the study. A changed subject or test rule needs the next revision. It must not change what retained evidence means.

Start the revision

In the Experiment view, choose Start this revision. The start contract explains the accepted request's identity. Inspect every returned arm, including refusals. Each accepted arm names its owner's Run. Read the Experiment again to check progress. The Console asks Run owners for their endings. Elapsed time alone cannot turn a running arm into a failure.

Compare the evidence

In /labs/comparisons, choose the study's objective version and candidate revisions. Review each candidate's evidence. Read the verdict beside its lineage: Experiment revision, composition digest, and owner Run references. Selecting a candidate records a preference and starts nothing. A Candidate Plan can store a proposal for review. The owning service still checks any later live change.

Checkpoints

The study's pins must match the intended test before start. After start, account for accepted, refused, and unstarted arms. In Experiment readback, check the revision beside those states. Confirm that every candidate uses the same objective version before comparison. A score needs its Case, version, and Run evidence to explain the ranking.

Success

The study has inspectable evidence for each candidate used in the Comparison. The verdict and lineage name the revisions and Runs that produced it. Keep refused or cancelled arms visible rather than presenting them as successful tests. A selected result remains a preference. A proposal remains a proposal. Neither is proof of a Model release, a live Workflow launch, or a production writeback.

When it fails

A study refusal must be read before another start. A partial start keeps the accepted work, so a failure is not proof that no subject ran.

SETUP_REQUIRED

The required study service is not configured. Ask the Workspace operator to repair that setup before starting again.

REVISION_CONFLICT

The revision changed after it was read. Reload the Experiment, review the changed pins, and select the intended revision again.

Recovery

A study retry of the same revision preserves accepted Runs and can retry refused arms. Repair the owner's refusal first. If a reported ending conflicts with an ending already kept, resolve it with that owner instead of rerunning.

To cancel accepted work, use the subject Run's owning service. A Simulation arm uses Simulation cancellation. Laboratory has no whole-Experiment cancel control. Read the owner's final state and retain its evidence. A cancelled Run is not a missing Run.

Next steps

  • Models and sandboxes explains the separate review needed to release a Model.
  • Agents & Workflows owns live execution after a study supports a change.
  • The API guide explains access for the documented Experiment create, start, and read contracts.