Skip to content
Merulan

Experiments and evaluations

Experiments pin a study's subjects and inputs; Evaluations score retained evidence and Comparisons explain the resulting order.

Experiments and evaluations test saved work in Labs & Modeling. A study pins what runs and scores the result. You can compare those scores against one goal.

Ownership

An Experiment belongs to Laboratory. It pins the objective version, subjects, Cases, method, and evaluation bundle. The objective states the goal and rules used to judge results. The subject's service owns its Run and outcome. The Lab records which Runs were accepted. A slow Run cannot be called failed based on elapsed time.

An Evaluation keeps scores, its gate result, and the evidence it read. A Comparison links each candidate to its study revision and goal. Its ranked results can thus be traced to the same rules.

In the Console, /labs/experiments lists studies and /labs/experiments/[id] opens one. /labs/evaluations inspects scores. /labs/comparisons compares candidates. /labs/cases/[caseId] opens the saved Case behind a test. Subject views at /labs/agents/[agentId] and /labs/workflows/[workflowId] keep the work tied to its owner.

Lifecycle

An Experiment edit creates the next revision. Start checks all pinned subjects before it starts any Run. The result lists accepted and refused arms. Each arm tests one subject. A retry of the same revision keeps accepted Runs and can retry refused arms.

Evaluation modes select the evidence to score. A prospective test scores a predicted result. A retrospective backtest uses past evidence. An actual-versus-predicted test checks a past prediction against what occurred. Saved Cases and scores keep their identity across tests.

Boundaries

An Experiment's saved policy evidence records an earlier decision. Current policy still applies at start. A Comparison requires candidates pinned to its goal's version. Evidence from another goal cannot enter the ranking.

An Evaluation does not deploy a Model or edit an Agent or Workflow. Its result informs a later choice. Models and sandboxes explains the release and scenario rules for that choice.

Failures

An Experiment refuses a missing, stale, or unsupported subject before dispatch. A later owner refusal can leave other arms accepted. Read all arms before retrying. One refusal does not mean nothing ran.

The Lab asks each owner for its Run's ending. If that result differs from an ending already kept, the Lab reports the conflict. The earlier evidence stays intact. A Comparison against a different goal version is refused.

Recovery

An Experiment recovery starts with readback. Inspect its exact revision and every arm. Repair the named input or access fault before starting that revision again. Keep the accepted Run references. Ask the subject owner to resolve a disputed ending.

For a changed test, create the next revision. To cancel, use the accepted Run's owning service and keep its final outcome. Run and compare a study follows that evidence through the decision.

  • Agents & Workflows owns the definitions and live work that a study can test.
  • Simulations explains sealed inputs and retained output for a Simulation subject.
  • The Console guide explains access before opening Experiments, Evaluations, or Comparisons under /labs.