Skip to content

Orchestration: methyl-workflow-run

Purpose

Choose the right entry point for staged validation on a workstation that has the methyl-gateway wheel installed. methyl-workflow-run is that wheel's local runner. This repository does not declare the console script. Production SamplePrep instances are started with methyl-study-start sample-prep-start, which calls finalize_instance_context and writes the instance to the database. See Start a SamplePrep instance.

The monolithic methyl-validation --stability/--freeze/--model path remains for transitional scripts and one-off recovery.

Workstation runner and legacy CLI

Workstation runner (gateway wheel) Legacy (transitional only)
Entry methyl-workflow-run methyl-validation --stability, --freeze, --model, …
Topology DomainProgram JSON (workflow_engine/domain/**/*.program.json) Hard-coded stage sequence inside the CLI
Parameters Profile actionConfig + site manifest + optional instance overrides Same keys, but resolved only through project path and CLI flags
Skip / resume Per-action .action_results/ manifests in the compiled graph --resume, --skip-detection, coarse stage flags
Production cluster Database instances claimed by methyl-worker through methyl-gateway. SamplePrep starts with methyl-study-start sample-prep-start. plan-runs / run-task file queue (legacy distributed MC)
When to use Workstation runs and smoke of a DomainProgram when the gateway wheel is installed Existing shell scripts, queue workers not yet migrated, narrow recovery

See also Dependency direction.

What you pass

Every canonical run needs three inputs:

  1. --program — workflow topology (which actions run, in what order, FOREACH loops, gates).
  2. --context-file — pipeline profile (*.profile.json) with merged actionConfig.
  3. --context — JSON with at least projectPath (study manifest on /work) and usually pipelineProfile.
source .venv/bin/activate
methyl-workflow-run \
  --program workflow_engine/domain/fixtures/mc_stability.program.json \
  --context-file workflow_engine/domain/profiles/mc_dmp_gene_fc.profile.json \
  --context '{"projectPath":"/work/projects/prostate-cancer/configs/project_Buffy_healthy_vs_PCa.json","pipelineProfile":"mc_dmp_gene_fc"}' \
  --parallel-workers 1

In production, point --program and --context-file at the promoted runtime bundle, e.g. /work/goliath/current/runtime-bundle/domain/... (see Deployment and distributed workflow).

Programs by scope

Pick a DomainProgram that matches how much of the lifecycle you need. Stage chapters (ch.05–09) describe artifacts and gates; this table maps them to programs.

Scope Example program Covers
MC stability only mc_stability.program.json, mc_stability_staged.program.json Centroid + detector loops, stability, freeze-readiness gate
Freeze only (after stability) pca1_5_freeze.program.json prepare_freeze_project, mapper, enricher, progression
Full study lifecycle study_validation_lifecycle.program.json Stability → freeze → model MC → selection → hold-out

Binary healthy-vs-cancer walkthrough: Tutorial: healthy vs. cancer stages.

Local development without a database

methyl-workflow-run \
  --program workflow_engine/domain/fixtures/mc_stability_smoke.program.json \
  --context-file workflow_engine/domain/profiles/mc_dmp.profile.json \
  --context '{"projectPath":"/work/projects/prostate-cancer/configs/project_Healthy_vs_PCa1-5-CG.json"}' \
  --stub-external

Validate compile/bindings before a long run: python workflow_engine/domain/checks/<study>/check_pipeline.py --program ....

Command presets

Copy-ready program/profile/context combinations:

bash scripts/workflow_presets.sh list
bash scripts/workflow_presets.sh show lifecycle

Set METHYL_RUNTIME_ROOT=/work/goliath/current/runtime-bundle on production workers to resolve bundle paths.

Legacy methyl-validation (when it still appears)

Use methyl-validation only when:

  • You must run an existing shell script or notebook that has not been migrated.
  • You need a narrow recovery command (e.g. --stability --skip-detection after MC iterations exist).
  • You operate the file-queue distributed path (plan-runs → run-task → aggregate-results) — see Distributed MethylValidation.

Do not start new studies with methyl-validation --stability as the primary entry. Migrate to a DomainProgram + profile (same actionConfig keys; different invocation).

# Legacy example — prefer methyl-workflow-run for new work
methyl-validation --project /work/projects/prostate-cancer/configs/project_Healthy_vs_PCa1-4-CG.json --stability

Package-level detail and CLI flag reference: packages/methylvalidation/docs/USAGE.md (documents legacy flags for parity with workflow actions).