Skip to content
Merulan

Experiments and evaluations

Experiments pin a study's subjects and inputs; Evaluations score retained evidence and Comparisons explain the resulting order.

Experiments and evaluations test saved work in Labs. A study pins what runs and scores the result. You can compare those scores against one goal.

Ownership

An Experiment belongs to Laboratory. It pins the objective version, subjects, Cases, method, and evaluation bundle. The objective states the goal and rules used to judge results. The subject's service owns its Run and outcome. The Lab records which Runs were accepted. A slow Run cannot be called failed based on elapsed time.

An Evaluation keeps scores, its gate result, and the evidence it read. A Comparison links each candidate to its study revision and goal. Its ranked results can thus be traced to the same rules.

In the Console, Labs › Evaluations opens /labs/evaluations to inspect scores. Studies list at /labs/experiments, and /labs/experiments/[id] opens one. /labs/comparisons compares candidates, while /labs/cases/[caseId] opens a saved Case. Subject views at /labs/agents/[agentId] and /labs/workflows/[workflowId] keep the work tied to its owner.

Labs › Workbenches opens independent, versioned canvases at /labs/workbenches. Create one at /labs/workbenches/new, inspect one at /labs/workbenches/[workbenchId], and read accepted invocations at /labs/runs/[runId]. Forecast questions from an installed forecasting Product appear under Analysis › Forecasts at /labs/forecasts. Without that Product, the list shows a setup-required state. The navigation placement does not change the service that owns each result.

Lifecycle

An Experiment edit creates the next revision. Start checks all pinned subjects before it starts any Run. The result lists accepted and refused arms. Each arm tests one subject. A retry of the same revision keeps accepted Runs and can retry refused arms.

Evaluation modes select the evidence to score. A prospective test scores a predicted result. A retrospective backtest uses past evidence. An actual-versus-predicted test checks a past prediction against what occurred. Saved Cases and scores keep their identity across tests.

Boundaries

An Experiment's saved policy evidence records an earlier decision. Current policy still applies at start. A Comparison requires candidates pinned to its goal's version. Evidence from another goal cannot enter the ranking.

An Evaluation does not deploy a Model or edit an Agent or Workflow. Its result informs a later choice. Models explains the release and scenario rules for that choice.

Failures

An Experiment refuses a missing, stale, or unsupported subject before dispatch. A later owner refusal can leave other arms accepted. Read all arms before retrying. One refusal does not mean nothing ran.

The Lab asks each owner for its Run's ending. If that result differs from an ending already kept, the Lab reports the conflict. The earlier evidence stays intact. A Comparison against a different goal version is refused.

Recovery

An Experiment recovery starts with readback. Inspect its exact revision and every arm. Repair the named input or access fault before starting that revision again. Keep the accepted Run references. Ask the subject owner to resolve a disputed ending.

For a changed test, create the next revision. To cancel, use the accepted Run's owning service and keep its final outcome. Run and compare a study follows that evidence through the decision.

Export and delegation

Experiments and evaluations compare exact subjects, methods, and results. Merulan retains the lineage of each artifact, exact dependencies, the evaluation contract, and the release decision. Tracking tools and compute may run outside Merulan. Merulan refuses a template or input reference that is absent, foreign, or expired. A notebook or tracker copy cannot show which inputs were allowed or whether two results compare.

  • Agents & Workflows owns the definitions and live work that a study can test.
  • Simulations explains sealed inputs and retained output for a Simulation subject.
  • The Console guide explains access before opening Experiments, Evaluations, or Comparisons under /labs.