Testing Framework

System-level testing for networked applications from Rust.

The testing framework deploys and controls processes, containers, and clusters. Tests can run several nodes over network connections for a bounded period. Application-specific configuration and clients stay outside the framework, so the runtime can be used with a key-value store, a Raft cluster, a message queue, or a blockchain.

Get Started

Scenarios

A declarative test is represented by a Scenario containing:

• Topology — the system under test (a uniform cluster, a composed application stack, or attached external nodes)
• Workloads — traffic and conditions that exercise the system
• Expectations — success criteria verified after execution
• Duration — the time window for the experiment

flowchart LR
    subgraph SC["Scenario"]
        T["topology<br/><small>the system under test</small>"]:::cl
        W["workloads<br/><small>drive traffic</small>"]:::sc
        EX["expectations<br/><small>verify outcomes</small>"]:::sc
        D["duration<br/><small>the run window</small>"]:::sc
    end
    SC --> RN["Runner<br/><small>deploy · run · evaluate · teardown</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The scenario runtime executes these parts in the same order for each declarative entry pattern. The entry pattern determines how the system is supplied.

Entry Patterns

flowchart LR
    A[Uniform managed cluster]:::cl --> S[Scenario]
    B[AppHost composed stack] --> S
    C[Attached / external nodes]:::cl --> S
    S:::sc --> R[Runner: workloads + expectations]:::sc
    M[ManualCluster] --> I[Imperative orchestration]
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

Three entry patterns use the scenario runtime:

1. Uniform managed cluster — the framework generates configs and launches N identical nodes from a topology. See Part IV.
2. AppHost composed stack — the app layer deploys heterogeneous components (processes, child clusters, in-process services) as one system and exposes typed handles to workloads. See Part II.
3. Attached and external nodes — the scenario targets clusters you already run, or plain URLs. See Existing and External Clusters.

ManualCluster is the imperative alternative. It provides direct start, stop, restart, and readiness operations without the scenario runner, including for step-driven BDD harnesses.

If you are not sure which to use, read Choosing an Entry Pattern.

Provided APIs

Declarative API

• Express tests as topology + workloads + expectations
• Reuse the same definition across local, Compose, and Kubernetes deployers
• Compose stacks from reusable application deployments

Application layer

• Deploy heterogeneous systems as one root AppDeployment
• Typed, named handles connect workloads to components
• Deterministic cleanup, including on partial-deployment failure

Runtime capabilities

• Per-cluster node control: restart nodes from workloads, portably
• Continuous observation: snapshots, history, and event streams of application state
• Telemetry: metrics, logs, and tracing endpoints

Operations

• Binary providers resolve node binaries from paths, env vars, builds, or downloads
• Reproducible deployments via seeds
• Artifact preservation for post-mortem debugging

Quick Example

use testing_framework_app::{AppHost, AppHostDeployer, AppScenarioBuilderExt as _, ClusterApp};

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_workload(KvClusterAccessible::new(3))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

This deploys a three-node key-value store cluster, runs a workload against it, and tears everything down. Swapping with_app for with_app_using(..., ComposeProvisioner::default()) or K8sClusterProvisioner runs the same scenario on Docker Compose or Kubernetes. The remaining chapters cover each part of this pattern in detail.

View the example apps

The Example Apps

The repository includes small applications under examples/ that exercise the framework APIs:

Some chapters also link to adopter repositories. The examples listed in this table run from this workspace.

Documentation Structure

Start with the Quickstart.

Quickstart

Run a complete multi-node test in one command.

Prerequisites

• Rust toolchain (the workspace pins its version via rust-toolchain.toml)
• Unix-like system (tested on Linux and macOS)
• For Compose examples: a running Docker daemon
• For Kubernetes examples: a reachable cluster context

No other setup. Example node binaries are resolved automatically; the kvstore example builds its node with Cargo on first run if no prebuilt binary is available.

Your First Test

git clone <this-repository>
cd <this-repository>
cargo run -p kvstore-examples --bin kvstore_basic_convergence

First run takes a few minutes (builds the framework and the kvstore-node binary).

What happens:

1. AppHost::scenario() creates an empty scenario that gets its whole system from application deployments.
2. with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3))) deploys a three-node kvstore cluster as local processes through the default LocalClusterProvisioner.
3. The write workload streams 300 keyed writes into the cluster while it runs.
4. The convergence expectation polls every node until all values agree, then the runner tears the cluster down.

What you should see:

• Three kvstore-node processes spawn with generated configs in per-run temporary directories
• The workload logs its write progress; the expectation reports convergence
• The command exits successfully and removes the temporary directories

The Code Behind It

The binary is short enough to read in full at examples/kvstore/examples/src/bin/basic_convergence.rs. Its core is:

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_run_duration(Duration::from_secs(30))
    .with_workload(
        KvWriteWorkload::new()
            .operations(300)
            .key_count(30)
            .rate_per_sec(30)
            .key_prefix("demo"),
    )
    .with_expectation(KvConverges::new("demo", 30).timeout(Duration::from_secs(25)))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The workload reaches the deployed cluster through a typed handle (RunContext is the object every workload receives at run time; see Part III):

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;
    let client = cluster.first_client().ok_or("kv cluster has no clients")?;

    put_value(&client, "demo-0", "value-0").await?;
    Ok(())
}

ClusterHandle is backend-neutral: swapping with_app for with_app_using(..., ComposeProvisioner::default()) or with_app_using(..., K8sClusterProvisioner) runs the same scenario on Docker Compose or Kubernetes without touching the workload.

The same pattern also runs in #[tokio::test] functions. The kvstore smoke test and the composition acceptance suite do this:

cargo test -p kvstore-examples --test local_smoke
cargo test -p multi-app-e2e

The latter uses a reusable fixture crate for a multi-application stack, workload, and expectation, then drives them from ordinary integration tests.

Where to Go Next

flowchart LR
    WL["workload<br/><small>enqueue 10 jobs</small>"]:::sc --> Q["queue cluster<br/><small>2 nodes</small>"]:::cl
    Q --> W["worker<br/><small>one process</small>"]:::pr
    W --> R["result store<br/><small>2 nodes</small>"]:::cl
    R --> EX["expectation<br/><small>10 results visible</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

let mut scenario = AppHost::scenario()                // ①
    .with_app(JobStackApp::new())                     // ②
    .with_run_duration(Duration::from_secs(10))       // ③
    .with_workload(EnqueueJobs::new(10))              // ④
    .with_expectation(AllJobsCompleted::new(10))      // ⑤
    .build()?;

let runner = AppHostDeployer
    .deploy(&scenario)                                // ⑥
    .await?;

runner.run(&mut scenario).await?;                     // ⑦

QueueScenario::nodes(5)
    .produce(400).rate_per_sec(40).done()
    .restart_nodes_randomly().every_secs(5, 15).done()
    .expect_converged(400).within_secs(60)
    .run_secs(120)
    .await?;

flowchart LR
    WL["produce<br/><small>400 jobs at 40/s</small>"]:::sc --> C
    subgraph C["queue cluster"]
        N0["node-0"]:::cl
        N1["node-1"]:::cl
        N2["node-2"]:::cl
        N3["node-3"]:::cl
        N4["node-4"]:::cl
    end
    RR["⚡ random restarts<br/><small>every 5–15s</small>"]:::pr -.-> C
    C --> EX["expect<br/><small>all 5 nodes converge at 400</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<QueueEnv>::new(QueueTopology::new(5)))
    .with_workload(
        QueueProduceWorkload::new()                       //  steady traffic
            .operations(400)
            .rate_per_sec(40)
            .payload_prefix("soak"),
    )
    .with_workload(
        ClusterRestartChaos::<QueueEnv>::new()            //  random node restarts
            .every_secs(5, 15)
            .cooldown_secs(60),
    )
    .with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
    .with_run_duration(Duration::from_secs(120))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;

runner.run(&mut scenario).await?;

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone;   // cluster shape
    type NodeClient: Clone + Send + Sync;            // how tests reach a node
    type NodeConfig: Clone + Send + Sync;            // per-node config type
}

flowchart TD
    U["Uniform cluster<br/><small>N identical nodes</small>"]:::cl --> S["Scenario"]:::sc
    A["Composed stack<br/><small>the job stack — line ②</small>"]:::sc --> S
    X["Attached / external<br/><small>clusters you already run</small>"]:::cl --> S
    S --> R["Runner<br/><small>one lifecycle for all three</small>"]:::sc
    M["ManualCluster<br/><small>managed nodes, you drive</small>"] -.->|bypasses the runner| C["step-by-step node control"]
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
    let queue = ctx
        .deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes))     // ①
        .await?;

    let results = ctx
        .deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
        .await?;

    let queue_url = queue.first_client().ok_or("queue cluster has no clients")?.base_url().clone();
    let results_url = results.first_client().ok_or("result store has no clients")?.base_url().clone();

    let worker = ctx
        .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))   // ②
        .await?;

    let stack = JobStackHandle { queue, results, worker };              // ③
    ctx.expose(stack.clone())?;

    Ok(stack)
}

struct JobStackHandle {
    queue: ClusterHandle<QueueEnv>,
    results: ClusterHandle<KvEnv>,
    worker: LocalProcessHandle<WorkerClient>,
}

let launch = LaunchSpec {
    binary: worker_binary_provider().resolve().await?,    // section 9
    args: vec!["--queue-url".to_owned(), queue_url.to_string(), /* … */],
    ..LaunchSpec::default()
};

let process = LocalProcessApp::new("job-worker", launch, endpoints, client)
    .with_readiness(|_, client| async move { client.wait_ready().await });

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))             // register activity
    .with_expectation(AllJobsCompleted::new(10))     // register the check
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;

runner.run(&mut scenario).await?;                    // TF invokes both

#[async_trait]
impl Workload<AppHostEnv> for EnqueueJobs {
    fn name(&self) -> &str {
        "enqueue_jobs"
    }

    async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let stack = ctx.require_app::<JobStackHandle>()?;
        let queue = stack
            .queue
            .first_client()
            .ok_or("queue cluster has no clients")?;

        for index in 0..self.count {
            let response: EnqueueResponse = queue
                .post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
                .await?;

            if !response.accepted {
                return Err(format!("queue rejected job {index}").into());
            }
        }

        Ok(())
    }
}

#[async_trait]
impl Expectation<AppHostEnv> for AllJobsCompleted {
    fn name(&self) -> &str {
        "all_jobs_completed"
    }

    async fn evaluate(&mut self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let stack = ctx.require_app::<JobStackHandle>()?;
        let clients = stack.results.clients();
        let deadline = Instant::now() + self.timeout;

        while Instant::now() < deadline {
            if all_results_are_visible(&clients, self.count).await? {
                if !stack.worker.is_running().await {
                    return Err("job worker stopped before evaluation".into());
                }

                return Ok(());
            }

            tokio::time::sleep(Duration::from_millis(200)).await;
        }

        Err(format!("job results did not converge within {:?}", self.timeout).into())
    }
}

flowchart TB
    R["runner sequences the test"]:::driver --> SETUP["system setup"]:::setup
    U["your Rust code / BDD steps<br/>sequence the test"]:::driver --> SETUP

    SETUP --> C["one uniform cluster"]:::shape
    SETUP --> S["composed stack"]:::shape
    S --> CC["uniform child clusters"]:::shape
    S --> P["standalone processes"]:::process

    C --> N["node bring-up<br/><small>topology → ports + peers → config → binary → start → readiness</small>"]:::engine
    CC --> N
    P --> SP["process bring-up<br/><small>launch settings → binary → start → readiness</small>"]:::engine

    N --> H["running resources<br/><small>clients · lifecycle control · reverse cleanup</small>"]:::runtime
    SP --> H

    classDef driver stroke:#9b6dd6,stroke-width:2.5px;
    classDef setup stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef shape stroke:#4a90d9,stroke-width:2.5px;
    classDef process stroke:#e08a3c,stroke-width:2.5px;
    classDef engine stroke:#777,stroke-width:2px;
    classDef runtime stroke:#4caf7d,stroke-width:2.5px;

#[tokio::test]
async fn drives_queue_cluster_without_a_scenario() -> Result<(), DynError> {
    let cluster = ManualCluster::<QueueEnv>::from_topology(QueueTopology::new(2));

    let node0 = cluster.start_node("node-0").await?.client;
    let node1 = cluster.start_node("node-1").await?.client;
    cluster.wait_network_ready().await?;

    enqueue(&node0, "manual-job").await?;
    wait_for_queue_len(&[node0, node1], 1).await?;

    cluster.restart_node("node-1").await?;
    cluster.wait_node_ready("node-1").await?;

    let restarted = cluster
        .node_client("node-1")
        .ok_or("node-1 client missing after restart")?;
    wait_for_queue_len(&[restarted], 1).await?;

    Ok(())
}

let mut deployment =
    DeployContext::<AppHostEnv>::new(AppHostTopology, NodeClients::default());
let stack = deployment.deploy(JobStackApp::new()).await?;

assert_eq!(stack.queue.node_count(), 2);
assert_eq!(stack.results.node_count(), 2);

let queue = stack.queue.first_client().ok_or("queue has no clients")?;
let results = stack.results.clients();
let worker = stack.worker.clone();

worker.restart().await?;
enqueue(&queue, "imperative-job").await?;
wait_for_completed_result(&results, "imperative-job").await?;

drop(deployment);                         // reverse cleanup for the whole stack
assert!(!worker.is_running().await);

flowchart TB
    T["1 · topology<br/><small>the test asks for three queue nodes</small>"]:::input
    V["2 · framework prepares node 1<br/><small>identity · reserved port · peer addresses</small>"]:::framework
    C["3 · application builds queue configuration<br/><small>node id · HTTP port · peers · sync interval</small>"]:::app
    B["4 · backend launches the node<br/><small>write config file · resolve binary · pass args and environment · start</small>"]:::backend
    H["5 · ready running resource<br/><small>typed client · lifecycle control · registered cleanup</small>"]:::runtime

    T --> V --> C --> B --> H

    POL["deployment policy<br/><small>readiness · retry · retained artifacts</small>"]:::policy -. "governs launch" .-> B
    POL -. "gates access" .-> H

    classDef input stroke:#777,stroke-width:2px;
    classDef framework stroke:#4a90d9,stroke-width:2.5px;
    classDef app stroke:#9b6dd6,stroke-width:2.5px;
    classDef backend stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef runtime stroke:#4caf7d,stroke-width:2.5px;
    classDef policy stroke:#c89b3c,stroke-width:2.5px;

fn build_cluster_node_config(
    node: &ClusterNodeView,
    peers: &[ClusterPeerView],
) -> Result<QueueNodeConfig, Error> {
    Ok(QueueNodeConfig {
        node_id: node.index() as u64,
        http_port: node.network_port(),
        peers: peers
            .iter()
            .map(|peer| QueuePeerInfo {
                node_id: peer.index() as u64,
                http_address: peer.authority(),
            })
            .collect(),
        sync_interval_ms: 500,
    })
}

fn local_process_spec() -> LocalProcessSpec {
    LocalProcessSpec::new("QUEUE_NODE_BIN")
        .with_binary_provider(queue_binary_provider())
        .with_rust_log("queue_node=info")
}

fn render_local_config(config: &QueueNodeConfig) -> Result<Vec<u8>, DynError> {
    yaml_node_config(config)
}

fn http_api_port(config: &QueueNodeConfig) -> u16 {
    config.http_port
}

flowchart TB
    C["typed per-node configuration"]:::app --> A["rendered per-node artifacts<br/><small>config file + any additional files</small>"]:::artifact
    A --> L["local backend<br/><small>write directly into the node working directory</small>"]:::local
    A --> S["container backends<br/><small>serve artifacts through cfgsync</small>"]:::container
    S --> F["cfgsync client in each container<br/><small>register · fetch · write files</small>"]:::container
    L --> N["start node binary"]:::process
    F --> N

    classDef app stroke:#9b6dd6,stroke-width:2.5px;
    classDef artifact stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef local stroke:#4a90d9,stroke-width:2.5px;
    classDef container stroke:#c89b3c,stroke-width:2.5px;
    classDef process stroke:#e08a3c,stroke-width:2.5px;

let node1 = cluster
    .start_node_with(
        "node-1",
        StartNodeOptions::<QueueEnv>::default().create_patch(|mut config| {
            config.sync_interval_ms = 50;
            Ok(config)
        }),
    )
    .await?
    .client;

let policy = DeploymentPolicy {
    readiness_enabled: true,
    readiness_requirement: HttpReadinessRequirement::AtLeast(2),
    retry_policy: Some(RetryPolicy::new(
        5,
        Duration::from_millis(500),
        Duration::from_secs(5),
    )),
    cleanup_policy: CleanupPolicy::new(true),
    ..DeploymentPolicy::default()
};

let cluster  = ctx.deploy_cluster(ClusterRequest::managed(deployment)).await?;
let attached = ctx.deploy_cluster(ClusterRequest::attached(existing)).await?;
let external = ctx.deploy_cluster(ClusterRequest::external(endpoints)).await?;

FallbackBinaryProvider::new([
    Arc::new(EnvBinaryProvider::new("MULTI_APP_JOB_WORKER_BIN")),
    Arc::new(BuildBinaryProvider {
        command: BuildCommand::new("cargo")
            .with_args(["build", "-p", "multi-app-job-worker", "--bin", "multi-app-job-worker"]),
        output_path: "target/debug/multi-app-job-worker".into(),
        working_dir: Some(workspace),
        lock_dir: None,
    }),
])

Part I — Mental Model

This part defines the main framework abstractions.

This part explains what the framework’s core types mean, how a scenario executes from build to teardown, and how to pick the right entry pattern for a given test before writing any code.

• Application, AppDeployment, and Environments — the three roles a “thing under test” can play
• Scenario Model and Lifecycle — what a scenario is and every phase it passes through
• Choosing an Entry Pattern — uniform cluster, composed stack, attached nodes, or manual control
• Ownership and Design Boundaries — what the framework owns versus what your application repository owns

Application, AppDeployment, and Environments

This chapter distinguishes the Application trait, the AppDeployment trait, and the concrete environment types that implement Application.

The Application Trait

Application is the contract between the scenario engine and whatever system you are testing. It bundles the backend-specific types the engine needs, without the engine ever knowing what your application does:

use testing_framework_core::scenario::Application;

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str; // default: "/"
}

The associated types and methods are:

• Deployment: the topology descriptor, i.e. how many nodes exist and how they relate.
• NodeClient: the typed client workloads use to talk to one node.
• NodeConfig: the per-node configuration your binary consumes.
• Client constructors: build_node_client turns deployer-provided NodeAccess into a client; external_node_client does the same for nodes the framework did not start. Both return an “unsupported” error by default. An environment must implement the operations it supports (Ownership and Design Boundaries).
• node_readiness_path: the HTTP path deployers probe during readiness gating.

An implementation of Application is called an environment. Everything generic in the framework (ScenarioBuilder<E>, Workload<E>, Expectation<E>, RunContext<E>) is parameterized over one.

Source: testing-framework/core/src/env.rs.

The AppDeployment Trait

Application describes a uniform node population. A system containing a cluster plus another process, or several different clusters, is represented through AppDeployment in testing-framework-app:

use testing_framework_app::{AppDeployment, AppHandle, DeployContext};

pub trait AppDeployment<E: Application, P = LocalClusterProvisioner>: Send + 'static {
    type Handle: AppHandle;

    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

An AppDeployment is a deployable unit: it consumes its description, prepares whatever it represents, and returns a typed runtime handle. AppHandle is a blanket implementation, so any Clone + Send + Sync + 'static type qualifies. The handle provides access and control; managed resources acquired through framework adapters are owned separately by scenario cleanup.

Deployments compose: inside deploy, the DeployContext lets a parent deployment call ctx.deploy(child) or ctx.deploy_and_expose(child), then ctx.expose(handle) to publish typed handles to workloads. See AppDeployment and DeployContext for the full context API.

An AppDeployment registered with .with_app(...) runs during scenario preparation. It participates in the scenario lifecycle; it does not replace that lifecycle.

Source: testing-framework/app/src/deployment.rs.

Concrete Environments

AppHostEnv: an environment without outer nodes

AppHostEnv is an environment with no outer nodes at all. Its topology, AppHostTopology, reports a node count of zero; its NodeClient and NodeConfig are both (); asking it for a node client is an error. It exists so that a scenario can be composed entirely from application deployments:

use testing_framework_app::{AppHost, AppScenarioBuilderExt};

let builder = AppHost::scenario()                              // ScenarioBuilder<AppHostEnv>, zero nodes
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)));   // apps provide all processes

The system is supplied by with_app deployments, and workloads access it through typed handles instead of outer node clients. See AppHost and with_app.

Source: testing-framework/app/src/host.rs.

KvEnv: a uniform node environment

The kvstore example shows a full environment for a real binary:

pub struct KvEnv;

impl Application for KvEnv {
    type Deployment = KvTopology;        // ClusterTopology
    type NodeClient = KvHttpClient;
    type NodeConfig = KvNodeConfig;

    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError> {
        Ok(KvHttpClient::new(access.api_base_url()?))
    }

    fn node_readiness_path() -> &'static str {
        "/health/ready"
    }
}

KvEnv additionally implements LocalBinaryApp (in examples/kvstore/testing/integration/src/local_env.rs) to tell the local deployer which binary to run, how to render per-node configs, and which port serves the HTTP API. The same environment type also backs AppDeployment presets like KvLocalApp, whose handle is a whole child cluster. Implementing Application walks through this in detail.

Source: examples/kvstore/testing/integration/src/app.rs.

How the Three Relate

graph TD
    SB["ScenarioBuilder&lt;E&gt;"] -->|"E: Application"| APP["Application<br/>(env contract)"]
    KV["KvEnv"] -.->|implements| APP
    AH["AppHostEnv<br/>(zero-node env)"] -.->|implements| APP
    AD["AppDeployment&lt;E, P&gt;"] -->|"deploys via"| DC["DeployContext&lt;E, P&gt;"]
    AD -->|returns| H["typed Handle"]
    SB -->|".with_app(...)"| AD
    SB:::sc
    H:::hd
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

Application and AppDeployment are not alternatives. Every scenario runs over AppHostEnv, and app deployments are registered inside it. A cluster brings its own environment type as the E of ClusterApp<E>, ClusterHandle<E>, and the workloads that drive it — one ClusterApp<KvEnv> for a uniform kvstore cluster, several apps for a heterogeneous stack.

Where to Go Next

• Scenario Model and Lifecycle: what a scenario is and how it runs.
• Choosing an Entry Pattern: which combination of these pieces fits your system.
• Part II — Composing Applications: the app layer in depth.
• Part IV — Uniform Clusters and Configuration: implementing an environment for your own binary.

Scenario Model and Lifecycle

A scenario records a topology, workloads, expectations, runtime settings, and deployment policy. The runner executes the phases described below.

What a Scenario Is

You assemble a scenario with ScenarioBuilder<E> (usually via AppHost::scenario()) and hand it to AppHostDeployer. The essential ingredients:

Because the whole plan is declared up front, build() can validate it and fail before any process is spawned.

Workloads implement Workload<E> (name(), init(...), async start(&self, ctx)); expectations implement Expectation<E> (start_capture, optional check_during_capture, evaluate). A workload can also contribute its own expectations; with_workload collects them automatically. Both receive the shared RunContext<E>, which carries the deployment descriptor, node clients, telemetry, and typed runtime extensions. See Workloads and Concurrency and Expectations and Evaluation.

The Lifecycle

flowchart TD
    B["build()"] --> D["AppHostDeployer.deploy(&scenario)"]
    D --> PX["prepare runtime extensions (with_app runs here)"]
    PX --> RG["provision clusters: spawn + readiness gating (retry per policy)"]
    RG --> RUN["runner.run(&mut scenario)"]
    RUN --> W["workloads start concurrently"]
    W --> CD["cooldown window"]
    CD --> EV["evaluate all expectations (aggregate failures)"]
    EV --> H["RunHandle"]
    H --> T["drop → cleanup guards"]
    W -- "failure" --> T
    EV -- "failure" --> T
    RUN:::sc
    W:::sc
    EV:::sc
    H:::hd
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

1. Build

build() finalizes the plan. It resolves the deployment from the topology provider (honoring with_deployment_seed) and calls init on every workload and expectation. Failures surface as ScenarioBuildError before anything is deployed.

Note: build() enforces a minimum run duration of 10 seconds and defaults the expectation cooldown to 10 seconds when you have not set one.

2. Deploy

AppHostDeployer.deploy(&scenario) prepares the runtime extensions (typed services prepared once per run and handed to workloads, see Runtime Extensions) and returns a Runner<E>. Registered with_app deployments run at this point: each cluster request is provisioned by its backend — nodes spawned, then readiness gating, where each node’s readiness probe (HTTP path or plain TCP, per the app) is retried until the request policy’s readiness requirement holds, with retry and backoff per DeploymentPolicy. Registering two extensions of the same type fails here with a “duplicate runtime extension type registered” error. Failure-path cleanup already applies at this stage through ownership: when deployment errors partway, partially deployed app resources are released as their handles drop, and spawned node processes stop when their process handles drop.

3. Run: workloads

runner.run(&mut scenario) first calls start_capture on every expectation, then spawns all workloads concurrently, each in its own task. Workload panics are caught and converted into workload errors instead of aborting the process. The run window lasts for the configured duration, during which the runner also ticks check_during_capture on every expectation once per second, so an expectation can fail during the run instead of waiting for final evaluation.

A workload returning early with Ok(()) is fine; the window keeps running while other workloads are still active. The duration is a maximum: once every workload has finished, the window ends early and cooldown begins. A workload error ends the run immediately with ScenarioError::Workload.

4. Cooldown and settle

When the duration elapses, workloads are not cut off abruptly. The runner keeps the run alive through a cooldown window derived from with_expectation_cooldown; managed clusters that actually granted node control get a 30-second minimum so restarted nodes and runtime extensions observe stabilized state — a managed cluster deployed without control does not impose the floor. Remaining workload tasks are then drained, and a short settle wait (at least 2 seconds when a cooldown is configured or any cluster granted node control) runs before evaluation.

5. Evaluation

Every expectation’s evaluate runs, including after another expectation fails. Failures are aggregated into one ScenarioError::Expectations report with one line per failed expectation.

6. Teardown

A successful run returns a RunHandle<E>. Teardown is guard-based. When the handle drops, its CleanupGuard chain runs, stopping node processes, aborting extension tasks, and executing app cleanup stacks (see Handle Ownership and Teardown). The same guards run on the failure path: any step that errors inside run triggers immediate cleanup before the error is returned, so failed runs do not leak managed processes or temp directories.

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_run_duration(Duration::from_secs(30))
    .with_expectation_cooldown(Duration::from_secs(5))
    .with_workload(KvWriteWorkload::new().operations(300))
    .with_expectation(KvConverges::new("demo", 30))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
let _handle = runner.run(&mut scenario).await?;
// dropping _handle tears the cluster down

Source: testing-framework/core/src/scenario/runtime/runner.rs and runtime/context.rs.

Errors by Phase

Where to Go Next

• Application, AppDeployment, and Environments: the type parameter behind ScenarioBuilder<E>.
• Choosing an Entry Pattern: the ways to reach this one lifecycle.
• Readiness, Retry, and Artifact Preservation: tuning the deploy phase.
• Part III — Scenario Runtime: workloads, expectations, and capabilities in depth.

Choosing an Entry Pattern

The framework has one composition model: every declarative scenario starts from AppHost::scenario() and registers its applications with .with_app(...) — one root application for a coupled stack, or several independent apps whose handle registries merge. What varies is what those applications are, where their clusters come from, and which backend provisions them. One imperative escape hatch — ManualCluster — sits outside the scenario runtime for tests where your code dictates every step.

One Model, Four Shapes

1. One uniform cluster. The degenerate — and most common — composed application is a single node cluster. ClusterApp<E> deploys N identical nodes of one Application and exposes a backend-neutral ClusterHandle<E> to workloads:

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_run_duration(Duration::from_secs(30))
    .with_workload(KvWriteWorkload::new().operations(300))
    .with_expectation(KvConverges::new("demo", 30))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

with_app uses the default LocalClusterProvisioner; with_app_using(app, provisioner) runs the same scenario on Docker Compose (ComposeProvisioner) or Kubernetes (K8sClusterProvisioner). This is cargo run -p kvstore-examples --bin kvstore_basic_convergence and its _compose_ / _k8s_ siblings.

2. Composed application stack. Your system is heterogeneous: several clusters, singleton processes, or both. Write one root AppDeployment that composes children through DeployContext and exposes typed handles that workloads retrieve with AppRunContextExt:

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())              // queue cluster + worker + result store
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))
    .with_expectation(AllJobsCompleted::new(10))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

A scenario accepts one with_app registration. A second registration fails at prepare time with a duplicate-runtime-extension error; compose multiple apps inside one root deployment instead (Composing Heterogeneous Stacks).

3. Attached and external clusters. The system already runs somewhere else: a staging network, a long-lived cluster, another team’s deployment. This is not a separate entry pattern — it is a different arm of the same per-cluster request. ClusterRequest::attached(existing) connects to a cluster the framework can discover (a compose project or a k8s label selector), ClusterRequest::external(nodes) wraps plain endpoints, and ClusterApp::with_external_nodes(...) mixes external endpoints into a managed cluster. Application::external_node_client turns each ExternalNodeSource into a typed client. Workloads and expectations are unchanged. See Existing and External Clusters.

4. ManualCluster: imperative control. Your code decides when nodes start, stop, and restart, step by step. ManualCluster gives you start_node, start_node_with(StartNodeOptions), stop_node, restart_node, wait_network_ready, wait_node_ready, and node_client, but no workloads, no expectations, no runner. It exists for local processes (testing_framework_runner_local::ManualCluster::from_topology) and Kubernetes (testing_framework_runner_k8s::ManualCluster::from_topology). See ManualCluster: Imperative Node Control.

Note: needing to restart nodes does not push you to ManualCluster. ClusterApp requests full node control by default, so its ClusterHandle exposes restart_node and friends to any workload (Cluster Control and Observability). Manual mode is also available inside the model: ClusterApp::with_start_mode(ClusterStartMode::OnDemand) prepares a cluster whose nodes your workloads start explicitly.

One Runtime, One Declarative Path

flowchart TD
    U["One uniform cluster<br/>ClusterApp&lt;E&gt;"] --> W["AppHost::scenario().with_app(...)"]
    A["Composed stack<br/>root AppDeployment"] --> W
    X["Attached / external<br/>ClusterRequest::attached / ::external"] --> W
    W --> S["Scenario"]
    S --> R["AppHostDeployer::deploy → Runner::run<br/>(one lifecycle, see Scenario Model)"]
    M["ManualCluster<br/>managed nodes, you drive"] -.->|"bypasses the runner"| C["imperative node control"]
    S:::sc
    R:::sc
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

Every declarative shape produces a Scenario<AppHostEnv> and uses the same lifecycle, so workloads and expectations are reused across shapes and backends whenever their required handles are available. ManualCluster uses the node-startup implementation without the scenario runtime. Its nodes remain framework-managed while test code controls the sequence.

Decision Table

Choosing by Example

“Three kvstore nodes, write traffic, convergence check.” One uniform cluster. KvEnv already models the node; ClusterApp owns the whole population. This is cargo run -p kvstore-examples --bin kvstore_basic_convergence.

“A queue cluster, a worker process, and a result-store cluster forming one pipeline.” Composed stack. One root AppDeployment deploys both clusters, wires the worker to them by URL, and exposes each handle plus a stack handle; a workload enqueues jobs and an expectation verifies the results. The multi-app-e2e acceptance test covers this shape; run it with cargo test -p multi-app-e2e.

“Run our smoke workload against the live staging network.” External. There is nothing to deploy: build the cluster from ClusterRequest::external(endpoints), let external_node_client build clients, and keep the exact same workloads and expectations you use locally.

“A Gherkin suite where each step starts or kills a node.” ManualCluster. The BDD runner owns sequencing, and its steps call start_node_with, stop_node, and wait_node_ready directly.

External example: logos-blockchain’s cucumber suite is a real example of the fourth pattern: Gherkin steps drive ManualCluster for dependency-ordered starts, restarts, and snapshot/restore flows, all in its own repository.

Where to Go Next

• Scenario Model and Lifecycle: the runtime every declarative shape converges on.
• Application, AppDeployment, and Environments: the types behind the model.
• Cluster Provisioning: the request/provisioner boundary that makes backends interchangeable.
• Ownership and Design Boundaries: what stays yours regardless of shape.

Ownership and Design Boundaries

This chapter lists the responsibilities of the framework and of an application repository.

The Boundary

The scenario engine never names a concrete application. Its only coupling point is the Application trait: a bundle of associated types (Deployment, NodeClient, NodeConfig) that the engine plumbs around generically. Everything that knows what your system is (its binary, its config format, its client, its notion of “healthy”) lives on your side of that trait.

What the Framework Owns

Process lifetime and working directories. Provisioners spawn node processes into per-run working directories, track PIDs, and stop everything on teardown. Artifact retention is policy (CleanupPolicy::preserve_artifacts), not something scenarios hand-roll.

Cleanup. Teardown is guard-based and automatic: cleanup guards chain and run in reverse registration order when the RunHandle drops, and the same guards run on the failure path. App-layer adapters register managed resources in a LIFO cleanup stack so dependants stop before dependencies, independently of exposed handle clones (Handle Ownership and Teardown).

Topology mechanics. Port allocation, peer wiring, node naming, and readiness gating with retry are all generic over E: Application. The engine asks your environment what to render and probe, never why.

Handle storage and lookup. DeployContext collects typed handles during preparation; AppRuntime carries them into the run; AppRunContextExt returns clones to workloads. Duplicate exposure of a type/name pair is an error, never a silent replacement.

What the Application Repository Owns

The kvstore example is the template. Its integration crate supplies, in its own repository:

• The binary and how to get it: a FallbackBinaryProvider chain that uses KVSTORE_NODE_BIN if set and otherwise builds kvstore-node with cargo (Binary Providers).
• Config: KvNodeConfig, built per node from the framework’s port/peer views and rendered to YAML.
• Client: KvHttpClient, constructed in Application::build_node_client.
• Readiness: node_readiness_path() returning /health/ready.
• Domain handles and presets: KvLocalApp and the workloads’ typed use of ClusterHandle<KvEnv>.
• Scenarios that mean something: write workloads, convergence expectations, runnable bins.

// Application side: domain knowledge, no orchestration.
fn node_readiness_path() -> &'static str {
    "/health/ready"
}

// Framework side: orchestration, no domain knowledge.
// It only ever sees E::NodeClient, E::NodeConfig, E::Deployment.

Sources: examples/kvstore/testing/integration/src/{app,local_env}.rs, testing-framework/app/src/lib.rs.

How the Boundary Is Enforced

Unsupported defaults. Application::build_node_client and external_node_client return an “unsupported” error by default. Capabilities are available only when the environment implements them.

Generic application types. There is no global list of known applications or framework config file naming their binaries. testing-framework-core compiles against E: Application, so it does not depend on adopter types or their dependencies. The same runtime can therefore be instantiated with kvstore, openraft_kv, nats, or an application from another repository.

Application-owned composition. Application repositories implement AppDeployment, compose children through DeployContext, and expose typed handles. The framework supplies the context, registry, and lifecycle without defining the application stack.

CI boundary check. scripts/run/check-boundaries.sh checks an application-side topology crate for framework-extension symbols (cfgsync, ComposeDeployEnv, K8sDeployEnv, runner-compose, runner-k8s). This detects backend dependencies in topology code. The compiler enforces the reverse direction because core crates do not reference concrete application types.

External example: the current boundary script targets logos-blockchain’s lb-topology crate (in its own checkout), which keeps that adopter’s topology code local/topology-focused. The pattern generalizes: point the same grep at your own integration crates.

Application-specific config formats and startup rules belong in the environment implementation or an AppDeployment, not in framework crates.

Where to Go Next

• Application, AppDeployment, and Environments: the trait that defines the boundary.
• Implementing Application: building your side of it.
• Framework vs Application Boundaries: the reference treatment in Part VI.
• Public Extension Points: the sanctioned ways to extend the framework itself.

Part II — Composing Applications

The app layer deploys heterogeneous systems as one unit and exposes typed handles to workloads.

Use this entry pattern when the system under test is not a single uniform cluster. A root AppDeployment deploys children such as processes, uniform child clusters, and in-process services. It exposes their handles, while the scenario runtime schedules test behavior and cleanup.

• AppHost and with_app — hosting a composed app inside a scenario
• AppDeployment and DeployContext — the deployment contract and its context
• Handle Ownership and Teardown — typed access, managed lifetime, and reverse cleanup
• One Binary: LocalProcessApp — the smallest building block
• Uniform Clusters: ClusterApp and ClusterHandle — a managed cluster as one component
• Composing Heterogeneous Stacks — the root-app pattern, end to end
• Backend Scope — what the app layer supports today

AppHost and with_app

AppHost creates a scenario whose system under test is supplied by application deployments instead of an outer managed node topology.

The core scenario engine models one Application and a uniform cluster of its nodes. For a composed stack containing a binary, an additional cluster, or several applications, start from AppHost::scenario() and register the stack with .with_app(...). Workloads, expectations, run duration, and teardown follow the lifecycle in Scenario Model and Lifecycle.

The Zero-Node Scenario

AppHost::scenario() returns a ScenarioBuilder<AppHostEnv> seeded with AppHostTopology:

Because the outer topology is empty, app deployments create the processes and clusters used by the run.

use testing_framework_app::{AppHost, AppHostDeployer, AppScenarioBuilderExt, ClusterApp};

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_run_duration(Duration::from_secs(5))
    .with_workload(KvClusterAccessible::new(3))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

with_app provisions clusters on local processes by default. with_app_using selects a backend provisioner per app instead — the same deployment moves to Docker Compose or Kubernetes without changing the rest of the scenario:

let mut scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<KvEnv>::new(KvTopology::new(3)),
        ComposeProvisioner::default(),
    )
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The app declaration remains backend-neutral: it receives published endpoints for test clients, internal endpoints for service dependencies, and portable per-service lifecycle handles.

The runnable kvstore_basic_convergence binary uses this structure:

cargo run -p kvstore-examples --bin kvstore_basic_convergence

How with_app Runs

AppScenarioBuilderExt::with_app(app) wraps your AppDeployment in an AppDeploymentFactory and registers it as a runtime extension factory, the same lifecycle hook covered in Runtime Extensions. Going through the extension mechanism ties managed deployment cleanup to the scenario lifetime and makes exposed handles available during the run.

flowchart LR
    B["with_app(app)"] --> F[AppDeploymentFactory]
    F -->|prepare| C[DeployContext]
    C -->|"deploy(root app)"| H["handles + cleanup"]
    H --> R[AppRuntime extension]
    R -->|require_app| W[Workloads]
    H:::hd
    R:::hd
    W:::sc
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

During scenario preparation the factory:

1. Clones your app (this is why the factory requires Clone) and builds a fresh DeployContext.
2. Runs the root deployment’s deploy, which may deploy and expose child apps.
3. Auto-exposes the returned root handle if the deployment did not expose one of that type itself (!ctx.contains::<A::Handle>()).
4. Transfers the handle registry and cleanup stack into an AppRuntime extension.

If any step fails, the partially built context is dropped and every resource deployed so far is released (see Handle Ownership and Teardown).

A scenario accepts multiple with_app registrations: every AppDeploymentFactory produces the same extension type (AppRuntime), and the runtime merges those registrations into one handle registry, chaining their cleanups in registration order. What must stay unique are the handles themselves — two apps exposing the same handle type and name fail during preparation with a duplicate-handle error, so distinct apps either expose distinct types or use named handles. Larger stacks whose parts depend on each other are still clearest as one root AppDeployment that deploys children in order, as shown in Composing Heterogeneous Stacks.

Retrieving Handles in Workloads

Workloads never see the deploy context. They retrieve exposed handles through AppRunContextExt, implemented on RunContext<E>:

use testing_framework_app::AppRunContextExt;

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;
    cluster.restart_node("node-0").await?;
    cluster.wait_node_ready("node-0").await?;
    Ok(())
}

Workloads normally use the require_* variants so that a missing handle produces a typed error containing the requested handle type.

Every retrieval clones the handle. Handles are normally small access values backed by Arc; scenario cleanup still determines managed resource lifetime.

Where to Go Next

• AppDeployment and DeployContext: implementing the deployment itself.
• One Binary: LocalProcessApp and Uniform Clusters: ClusterApp and ClusterHandle: the two built-in building blocks.
• Backend Scope: what each backend provisioner supports.

AppDeployment and DeployContext

Application repositories implement AppDeployment for deployable components. The implementation uses DeployContext to deploy children, expose handles, and register managed resources with scenario cleanup.

The framework runs deployments, stores handles, and performs teardown without defining application binaries or clients. The application crate decides which components start and which typed handles workloads receive.

The Trait Contract

#[async_trait]
pub trait AppDeployment<E: Application, P = LocalClusterProvisioner>: Send + 'static {
    type Handle: AppHandle;

    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

The trait has these properties:

• deploy consumes self. A deployment value describes one preparation attempt and returns its runtime access handle.
• The handle is typed. Handle can be any Clone + Send + Sync + 'static type. Managed lifetime is registered separately; see Handle Ownership and Teardown.
• Clone is required by the factory. with_app needs A: AppDeployment<E> + Clone + Sync because AppDeploymentFactory clones the description on each prepare. Deployment structs should contain configuration such as node counts, ports, and paths rather than live resources.

A minimal implementation, from the kvstore example:

// examples/kvstore/testing/integration/src/app.rs
#[derive(Clone)]
pub struct KvLocalApp {
    deployment: KvTopology,
}

#[async_trait]
impl AppDeployment<AppHostEnv> for KvLocalApp {
    type Handle = ClusterHandle<KvEnv>;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        ctx.deploy_cluster(ClusterRequest::<KvEnv>::managed(self.deployment))
            .await
    }
}

DeployContext API

One context belongs to one scenario preparation. It carries the active cluster provisioner, outer deployment and clients, exposed handles, and a cleanup stack. Routing every managed child through this context registers cleanup as soon as the resource is acquired, including when deployment fails partway.

deploy does not expose its returned handle. Use it when only the parent needs the child handle. Use deploy_and_expose when workloads should also be able to request the child directly. Both expose and expose_named return AppDeployError::DuplicateHandle if the type or type/name pair is already registered.

Nested Deployments

A deployment composes children by calling ctx.deploy(...) or ctx.deploy_and_expose(...) on other AppDeployment values. The parent decides what is visible:

#[async_trait]
impl AppDeployment<TestEnv> for ParentApp {
    type Handle = ParentHandle;

    async fn deploy(self, ctx: &mut DeployContext<TestEnv>) -> Result<Self::Handle, DynError> {
        let child = ctx.deploy(ChildApp).await?; // child handle NOT exposed
        let parent = ParentHandle { child };

        ctx.expose(parent.clone())?; // only the parent is visible

        Ok(parent)
    }
}

Workloads can then require ParentHandle but not ChildHandle: the child stays an implementation detail. Its managed resources remain registered with scenario cleanup whether or not the returned handle is exposed or embedded. Expose the child too when workloads legitimately need it.

flowchart TD
    Root[Root AppDeployment] -->|deploy| C1[Child A]
    Root -->|deploy_and_expose| C2[Child B]
    Root -->|expose| RH[Root handle]
    C2 --> BH[Child B handle]
    RH --> W[Workloads]
    BH --> W
    RH:::hd
    BH:::hd
    W:::sc
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

Non-Managed Clusters: attached and external

For AppHost scenarios, deployment() is the empty AppHostTopology and node_clients() is empty; everything lives in your handles. A deployment that should use nodes the framework does not own builds the corresponding ClusterRequest instead of a managed one:

// attach to a live deployment described by ExistingCluster
let cluster = ctx
    .deploy_cluster(ClusterRequest::<KvEnv>::attached(existing))
    .await?;

// or build clients for plain external endpoints
let cluster = ctx
    .deploy_cluster(ClusterRequest::<KvEnv>::external(nodes))
    .await?;

Neither request launches nodes; both return a ClusterHandle with typed access to the nodes, and cleanup does not stop resources the framework did not start. See External and Existing Clusters.

Root-Handle Auto-Exposure

After the root deployment returns, AppDeploymentFactory checks ctx.contains::<A::Handle>(). If the root handle type is not already exposed, the factory exposes the returned handle as the default for its type. So:

• A simple root app can just return Ok(handle) and workloads can require_app::<Handle>() with no explicit expose.
• A root app that already exposed its own handle (like the stack apps in Composing Heterogeneous Stacks) is left alone, so there is no duplicate error.

See Also

• AppHost and with_app: how a deployment gets registered and prepared.
• Handle Ownership and Teardown: what exposure means for resource lifetime.
• One Binary: LocalProcessApp, Uniform Clusters: ClusterApp and ClusterHandle: ready-made deployments to compose.

Handle Ownership and Teardown

Handles provide typed access to deployed applications. The scenario runtime owns managed resource lifetime separately through a cleanup stack.

Cloning a handle preserves access to its shared state, but does not extend a process or cluster beyond the run that created it.

Typed and Named Handles

The registry keys every handle by concrete type plus name (TypeId and a string). An empty name is the default handle for that type.

Duplicate exposure is an error rather than a replacement. Use distinct names when a scenario exposes multiple values of one handle type, then retrieve them with app_named or require_app_named.

Missing handles are typed runtime errors. require::<T>() and require_named::<T>(name) return AppDeployError::HandleMissing with the requested Rust type and instance name.

What a Handle Means

pub trait AppHandle: Clone + Send + Sync + 'static {}

impl<T> AppHandle for T
where
    T: Clone + Send + Sync + 'static,
{}

An app handle can be a client, a control surface, or a domain aggregate such as JobStackHandle. Retrieval clones it so workloads can use it without borrowing the registry.

The registry lookup uses TypeId. Requesting the wrong concrete type is therefore a runtime miss, not a compile-time error. Prefer specific handle types or domain newtypes over primitives whose role is unclear.

Cloneability is about access, not ownership of the deployment. LocalProcessHandle clones share process state and controls; ClusterHandle clones share clients and control adapters. Scenario cleanup remains authoritative for managed lifetime.

Managed Lifetime

Every framework adapter that acquires a managed resource registers a cleanup guard immediately. DeployContext collects those guards in acquisition order and transfers the stack to the scenario runtime after successful preparation.

flowchart LR
    D["deploy child"] --> G["register cleanup guard"]
    G --> H["return and optionally expose handle"]
    H --> R["scenario runs"]
    R --> C["cleanup stack: last acquired, first released"]

This produces two parallel structures:

The cleanup stack decides when managed resources stop. A handle clone retained outside the registry does not postpone cleanup; after cleanup, operations on a LocalProcessHandle fail because the run no longer owns the process.

Dependency-Ordered Teardown

Deploy dependencies before dependents:

let queue = ctx.deploy_and_expose(QueueLocalApp::nodes(2)).await?;
let results = ctx.deploy_and_expose(KvLocalApp::nodes(2)).await?;
let worker = ctx
    .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))
    .await?;

Each deployment registers cleanup as soon as it acquires its resource. LIFO cleanup therefore stops the worker first, then the result store, then the queue. The order is independent of which handles the final JobStackHandle embeds or how many clones workloads retain.

Exposure order usually follows acquisition order, but it is not the ownership mechanism. Expose a handle when test code needs to find it; register cleanup when the framework acquires a managed resource.

Partial-Deployment Failure

If deployment fails halfway through, dropping DeployContext runs every cleanup guard already registered. The same LIFO rule applies, so successfully started dependents stop before their dependencies even though no scenario runner was created.

Readiness belongs inside deployment for the same reason. LocalProcessApp::with_readiness stops its just-started process if the check fails, while the context cleans up all earlier children.

Custom AppDeployment implementations should acquire managed resources through framework adapters such as LocalProcessApp and deploy_cluster. A raw process started directly by application code has no cleanup guard unless that code implements and registers an adapter.

Manual Control During a Run

Automatic teardown does not prevent explicit control. A workload can call stop, start, or restart on a process handle, or the corresponding node methods on a cluster handle. Cleanup remains registered and idempotently closes whatever is still active when the run ends.

Managed applications therefore support both properties:

• test code can deliberately change runtime state;
• every exit path still has a final owner that cleans up.

Keeping Artifacts

Managed cleanup normally removes generated working directories. Use LocalProcessApp::keep_tempdir(true) or LocalProcessHandle::keep_tempdir() for a process. Cluster artifact retention is controlled by the cluster’s deployment policy described in Readiness, Retry, and Artifact Preservation.

See Also

• AppDeployment and DeployContext: where children, handles, and cleanup are assembled.
• One Binary: LocalProcessApp: a managed process and its control handle.
• Shared Cluster Provisioning: cluster handles across ownership modes.

One Binary: LocalProcessApp

LocalProcessApp<C> deploys one local binary with a typed client, without modeling it as a node topology.

Application code supplies the launch files, client type, and readiness check. The framework manages the process lifetime, working directory, and teardown. This is used for third-party infrastructure such as a message broker or database, and for singleton services such as a sequencer or indexer inside a composed stack.

Construction

LocalProcessApp::new(label, launch, endpoints, client)

LaunchSpec (from testing_framework_runner_local) is a plain launch plan:

NodeEndpoints describes where the process listens: an api: SocketAddr plus extra_ports keyed by NodeEndpointPort (TestingApi, Network, or Custom(String)). Build one with NodeEndpoints::from_api_port(port) and insert_port.

Endpoints are declared, not allocated. The generic process layer does not select ports. The launch configuration and the supplied endpoints must use the same values.

Builder Options

If the readiness closure fails, deployment returns an error, stops the new process, and cleans up children deployed earlier (see Handle Ownership and Teardown).

Example: A Single nats-server Process

The nats example normally runs as a uniform cluster, but its NatsClient works just as well against one broker started as a process app:

use std::time::Duration;

use nats_runtime_ext::NatsClient;
use testing_framework_app::LocalProcessApp;
use testing_framework_runner_local::{LaunchSpec, NodeEndpoints};

let launch = LaunchSpec {
    binary: std::env::var("NATS_SERVER_BIN")?.into(),
    args: vec!["-p".into(), "4222".into(), "-m".into(), "8222".into()],
    ..LaunchSpec::default()
};

let client = NatsClient::new(
    "nats://127.0.0.1:4222".to_owned(),
    "http://127.0.0.1:8222".parse()?,
);

let broker = LocalProcessApp::new("nats", launch, NodeEndpoints::from_api_port(8222), client)
    .with_readiness(|_endpoints, client| async move {
        for _ in 0..50 {
            if client.is_healthy().await.unwrap_or(false) {
                return Ok(());
            }
            tokio::time::sleep(Duration::from_millis(100)).await;
        }
        Err("nats-server did not become healthy".into())
    });

broker is an AppDeployment for any environment, so a root deployment composes it like any other child:

let nats = ctx.deploy_and_expose(broker).await?;

The Handle: LocalProcessHandle

deploy returns LocalProcessHandle<C>. Clones share access to the same process state, while scenario cleanup owns the process lifetime. Cleanup stops the process even if a handle clone still exists.

A workload retrieves the handle like any other (see AppHost and with_app):

let broker = ctx.require_app::<LocalProcessHandle<NatsClient>>()?;
broker.restart().await?;
assert!(broker.is_running().await);

Tests can use start, restart, and stop for process lifecycle and fault injection. These operations fail after scenario cleanup has closed the managed resource.

See Also

• AppDeployment and DeployContext: composing a process app under a root deployment.
• Composing Heterogeneous Stacks: mixing single processes with child clusters.

Uniform Clusters: ClusterApp and ClusterHandle

ClusterApp<E> deploys a uniform cluster of an application’s nodes as one child of a composed stack, and ClusterHandle<E> is the backend-neutral handle it exposes.

For N identical nodes of one binary with peer wiring and per-node clients, register a ClusterApp with with_app (local processes) or with_app_using (Compose or Kubernetes). A scenario over one uniform cluster is simply the composition with a single child.

For local runs the environment E must implement LocalDeployerEnv (config rendering, ports, process spec; see Local Deployer). That work is the same whether the cluster is the whole system or one component of a larger stack, so a cluster env is reusable in both shapes unchanged.

Deploying a Cluster

The prebuilt ClusterApp<E> covers the common case directly on the builder:

use testing_framework_app::{AppHost, AppScenarioBuilderExt as _, ClusterApp};

let builder = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)));

ClusterApp builds a managed ClusterRequest for the deployment: nodes start eagerly, full node control is requested, and the backend registers cleanup with the scenario. Optional modifiers adjust the request:

Inside a custom AppDeployment, the same operation is available as DeployContext::deploy_cluster, which launches every node described by the deployment (node-0, node-1, …), waits for readiness per policy, registers cleanup, and returns the handle:

// examples/kvstore/testing/integration/src/app.rs
#[async_trait]
impl AppDeployment<AppHostEnv> for KvLocalApp {
    type Handle = ClusterHandle<KvEnv>;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        ctx.deploy_cluster(ClusterRequest::<KvEnv>::managed(self.deployment))
            .await
    }
}

Scenario cleanup stops any remaining nodes independently of handle clones.

The Handle API

Node names follow the node-{index} convention used at startup. ClusterHandle<E> is backend-independent: the same methods work whether the cluster was provisioned on local processes, Compose, or Kubernetes, and control methods error at call time when the backend did not grant the corresponding control.

Per-node control is provided by the cluster handle; nothing needs to be enabled on the scenario for a workload to restart a cluster node.

Worked Example: kvstore Convergence

The kvstore_basic_convergence bin runs a three-node kv cluster as an app and exercises convergence under write load:

// examples/kvstore/examples/src/bin/basic_convergence.rs
let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
    .with_run_duration(Duration::from_secs(30))
    .with_workload(KvClusterAccessible::new(3))
    .with_workload(KvWriteWorkload::new().operations(300).key_count(30))
    .with_expectation(KvConverges::new("demo", 30))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

Workloads require the cluster handle and drive it directly:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;

    ensure_cluster_shape(&cluster, self.expected_nodes)?;
    put_value(&cluster, "before-restart").await?;
    cluster.restart_node("node-0").await?;
    cluster.wait_node_ready("node-0").await?;
    put_value(&cluster, "after-restart").await?;

    Ok(())
}

put_value writes through cluster.first_client(); ensure_cluster_shape checks node_count(), clients(), node_client("node-0"), and node_pid("node-0"). Run it with:

cargo run -p kvstore-examples --bin kvstore_basic_convergence

The kvstore environment resolves its node binary through a fallback provider chain, so this example does not require a manually configured binary path (see Binary Providers).

Exposing the Cluster to Workloads

ClusterApp returns the raw ClusterHandle<E> as its handle, and the factory auto-exposes it, so workloads request ClusterHandle<KvEnv> directly. In a composed stack you can either expose the raw cluster handle (as the job stack does), wrap it in a domain newtype (StoreHandle) for clearer requirements, or use named handles when two child clusters share an environment type (see Composing Heterogeneous Stacks).

See Also

• One Binary: LocalProcessApp: the single-process counterpart.
• Backend Scope: what each backend provisioner supports.

Composing Heterogeneous Stacks

A root AppDeployment deploys the components, passes dependency addresses between them, and exposes typed handles.

A scenario accepts one with_app registration (see AppHost and with_app), so the root deployment composes its children. It exposes component handles needed by workloads and may also expose an aggregate stack handle. The examples/multi_app fixture contains a queue cluster, job-worker process, and kv result-store cluster in one job-processing pipeline.

The Root App

// examples/multi_app/fixture/src/lib.rs
#[derive(Clone)]
struct JobStackApp {
    queue_nodes: usize,
    result_nodes: usize,
}

impl JobStackApp {
    fn new() -> Self {
        Self {
            queue_nodes: 2,
            result_nodes: 2,
        }
    }
}

#[async_trait]
impl AppDeployment<AppHostEnv> for JobStackApp {
    type Handle = JobStackHandle;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        let queue = ctx
            .deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes))
            .await?;
        let results = ctx
            .deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
            .await?;

        let queue_url = queue
            .first_client()
            .ok_or("queue cluster has no clients")?
            .base_url()
            .clone();
        let results_url = results
            .first_client()
            .ok_or("result store has no clients")?
            .base_url()
            .clone();
        let worker = ctx
            .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))
            .await?;

        let stack = JobStackHandle { queue, results, worker };
        ctx.expose(stack.clone())?;

        Ok(stack)
    }
}

The example establishes these relationships:

• Children are deployed through the context (deploy_and_expose), so each cluster and the worker process are owned by the runtime for the whole run.
• Dependencies are constructor arguments. The worker receives the queue and result-store URLs from the already-deployed clusters, so the root deployment shows the dependency graph.
• Both levels are exposed: each component handle and the aggregate JobStackHandle, allowing workloads to request the smallest handle they need.

flowchart TD
    Root[JobStackApp] --> Q["queue cluster x2<br/>ClusterHandle&lt;QueueEnv&gt;"]
    Root --> R["result store x2<br/>ClusterHandle&lt;KvEnv&gt;"]
    Q --> W["job worker<br/>LocalProcessApp"]
    R --> W
    Q --> St[JobStackHandle]
    R --> St
    W --> St
    Q:::cl
    R:::cl
    W:::pr
    St:::hd
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

Workloads Require What They Need

Each workload asks for exactly the handles it uses, the whole stack or one component:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let stack = ctx.require_app::<JobStackHandle>()?;
    let queue = stack.queue.first_client().ok_or("queue cluster has no clients")?;

    for index in 0..self.count {
        let response: EnqueueResponse = queue
            .post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
            .await?;
        if !response.accepted {
            return Err(format!("queue rejected job {index}").into());
        }
    }
    Ok(())
}

Assembling and running the scenario is unchanged from any other AppHost run:

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))
    .with_expectation(AllJobsCompleted::new(10))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

cargo test -p multi-app-e2e

Wiring Dependencies Between Components

Pass dependencies through constructors. Deploy the dependency first and pass its handle or address into the dependent’s constructor, as JobWorkerApp::new(queue_url, results_url) does above. This records the dependency graph and acquisition order in the root deployment. A child can call ctx.require::<T>(), but then it depends on another deployment having exposed T earlier. If that did not happen, deployment fails at run time with HandleMissing.

The same rule applies to process-level wiring. The job worker is a LocalProcessApp whose LaunchSpec receives the queue and store URLs as command-line arguments: deploy the dependency, read its client’s base_url(), and feed the address into the process. Do not have the process guess.

Use named handles for two instances of one type. The registry allows one default handle per concrete type; a second expose of the same type is a duplicate error (see Handle Ownership and Teardown). Two kv clusters in one stack therefore need names:

ctx.expose_named("primary", primary)?;
ctx.expose_named("replica", replica)?;

// in the workload:
let primary = ctx.require_app_named::<ClusterHandle<KvEnv>>("primary")?;

Expose components as well as the stack when both are used. A workload that touches one component can request its handle directly, while stack-level workloads can request the aggregate handle.

See Also

• AppDeployment and DeployContext: the composition API in detail.
• Uniform Clusters: ClusterApp and ClusterHandle: the child clusters used here.
• Backend Scope: where composed stacks can run.

Backend Scope

Every scenario is composed from application deployments, but the resources an application can ask its backend for differ. This chapter lists what each backend provisions and the APIs missing from the container backends.

What Works Where

The cluster row is the uniform path: ClusterApp<E> runs on all three backends by passing the matching provisioner to with_app_using (LocalClusterProvisioner is the with_app default). Compose is unique in serving both contracts from one type — ComposeProvisioner implements ClusterProvisioner<E> and ContainerStackProvisioner. Every cluster and container stack runs in its own compose project attached to one shared session network per provisioner, so clusters and container services reach each other by compose service name. Kubernetes has a cluster provisioner only: there is no k8s container-stack provisioner.

An app deployment that provisions nothing — one that only wraps clients or other handles — works on every backend because it never touches the provisioner. LocalProcessApp remains a local adapter for third-party or singleton binaries. Containerized apps describe images, commands, generated files, ports, readiness, and explicit restart policy through ContainerServiceSpec; stack containers default to no automatic restart so crashes remain observable.

The kvstore and OpenRaft examples keep dedicated bins per backend (kvstore_compose_convergence, kvstore_k8s_convergence, openraft_kv_compose_failover, openraft_kv_k8s_failover); see Compose Backend and Kubernetes Backend.

Remaining Kubernetes Gap

The container contract is independent from application composition and backend implementation. A Kubernetes ContainerStackProvisioner still needs to map it to a shared namespace, workload resources, Services, generated ConfigMaps, readiness, stable runner access, per-service lifecycle control, and cleanup. That should not require changes to containerized app deployments or workloads: JobStackContainerApp in examples/multi_app is already generic over any ContainerStackProvisioner.

Choosing a Shape Today

flowchart TD
    Q{System under test} -->|one uniform cluster| U[ClusterApp]
    Q -->|composed stack| A[root AppDeployment]
    U --> B{Backend provisioner}
    B --> L1[Local]
    B --> C1[Compose]
    B --> K1[Kubernetes]
    A --> AB{App resource model}
    AB -->|local processes and clusters| L2[Local]
    AB -->|container services| C2[Compose]
    AB -->|container services| K2[Kubernetes: pending]

• Composed stacks: use Local or Compose. Local adapters manage binaries, processes, and child clusters. Compose manages portable container declarations, per-service lifecycle operations, and node clusters, each in its own project on one shared session network per provisioner.
• Single-cluster scenarios: use any backend. The same ClusterApp scenario runs locally, on Compose, or on Kubernetes, subject to the capability matrix.
• Keep app handles backend-independent. The multi_app example runs the same workload and expectation over local processes and compose containers; only its resource declaration and provisioner differ.

Keep Workloads Backend-Independent

Write workloads against typed handles and clients, not against backend details. A workload that requires a ClusterHandle<KvEnv> does not care which provisioner produced it:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let store = ctx.require_app::<ClusterHandle<KvEnv>>()?; // no backend visible here
    let client = store.first_client().ok_or("store has no clients")?;
    put_value(&client, "scope-check", "ok").await?;
    Ok(())
}

Backend-specific changes remain in provisioners. Workloads and expectations continue using the same typed app handles across local, compose, and k8s runs.

Note: containerized ContainerServiceHandles provide start, stop, restart, wait_ready, and is_running. Node clusters expose the richer ClusterHandle API. Kubernetes must implement the same service-handle lifecycle before its container-stack row becomes supported.

See Also

• Backend Capability Matrix: the full feature-by-backend table.
• Cluster Provisioning: the request/provisioner boundary.
• AppHost and with_app: the entry point this scope applies to.

Part III — Scenario Runtime

These chapters describe what happens while a scenario runs: traffic, verification, capabilities, and observation.

The same runtime serves all three declarative entry patterns: uniform clusters, composed app stacks, and attached external nodes.

• Workloads and Concurrency — driving the system under test
• Expectations and Evaluation — verifying outcomes
• The Verb Layer — concise domain actions over the explicit builder API
• Scenario Capabilities — runtime features such as node control, per cluster
• Chaos and Controlled Failure — restarts and failover from workloads
• Runtime Extensions — typed scenario-lifetime services
• Continuous Observation — snapshots, history, and event streams for test logic
• Telemetry and External Observability — metrics, logs, and tracing

Workloads and Concurrency

Workloads describe the activity a scenario generates: every workload runs as its own concurrent task against the shared RunContext, and the runner decides when the run window ends.

The Workload Trait

A workload is any type implementing Workload<E> from testing-framework-core (testing-framework/core/src/scenario/workload.rs):

use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext, Workload};

#[async_trait]
pub trait Workload<E: Application>: Send + Sync {
    fn name(&self) -> &str;

    fn expectations(&self) -> Vec<Box<dyn Expectation<E>>> {
        Vec::new()
    }

    fn init(
        &mut self,
        _descriptors: &E::Deployment,
        _run_metrics: &RunMetrics,
    ) -> Result<(), DynError> {
        Ok(())
    }

    async fn start(&self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

The trait methods are:

• name identifies the workload in logs and failure reports.
• expectations lets a workload attach its own checks. with_workload collects them into the scenario alongside explicitly added expectations (see Expectations and Evaluation).
• init runs synchronously at build() time, before anything is deployed. It receives the resolved deployment descriptors and the RunMetrics (run duration). A failing init aborts the build with a WorkloadInit error.
• start is the async body. It runs once per scenario run and must return when its work is done.

The runner schedules every workload and applies the same concurrency, panic capture, and run-window behavior described below.

Register workloads on any builder with .with_workload(w) or .with_workload_boxed(boxed).

How the Runner Executes Workloads

The runner (testing-framework/core/src/scenario/runtime/runner.rs) drives a run in fixed phases:

flowchart LR
    P[start_capture<br/>expectations]:::sc --> W[Workload window<br/>run_duration]:::sc
    W --> C[Cooldown window]:::sc
    C --> D[Drain remaining<br/>workloads]:::sc
    D --> S[Settle wait]:::sc
    S --> E[Evaluate<br/>expectations]:::sc

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The runner uses the following concurrency rules:

• All workloads run concurrently. Each workload is spawned into its own Tokio task via a JoinSet; there is no ordering between them.
• Panics become errors. A panicking workload does not abort the process; the panic is caught and reported as workload panicked: <message>.
• One failure fails the run. The runner joins workload tasks as they finish. The first workload that returns Err (or panics) ends the run immediately with ScenarioError::Workload; expectations are not evaluated.
• Finishing early ends the window early. If every workload returns Ok before with_run_duration elapses, the workload phase completes without waiting out the timer.
• The run duration sets the maximum workload window but does not cancel workloads. When every workload finishes early, the window ends early and cooldown begins. When the timer expires while workloads are still running, the runner keeps driving them through the cooldown window and then waits for them to finish (drain_workloads). A workload that never returns blocks the run indefinitely.

Treat with_run_duration as the guaranteed run window, not as a workload timeout. A long-running workload should bound its own work, either by operation count or by reading ctx.run_duration() and stopping at the deadline.

After the workload window, managed deployments get a cooldown window (minimum 30 seconds when the framework owns the node lifecycle) plus a short settle wait so runtime extensions catch up before evaluation. Both are tuned with with_expectation_cooldown; see Expectations and Evaluation.

Worked Example: a Key/Value Write Workload

The kvstore example ships KvWriteWorkload (examples/kvstore/testing/workloads/src/write.rs), a rate-limited writer over the node HTTP clients:

use async_trait::async_trait;
use kvstore_runtime_ext::KvEnv;
use testing_framework_app::{AppHostEnv, AppRunContextExt as _};
use testing_framework_core::scenario::{ClusterHandle, DynError, RunContext, Workload};

#[async_trait]
impl Workload<AppHostEnv> for KvWriteWorkload {
    fn name(&self) -> &str {
        "kv_write_workload"
    }

    async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;
        let Some(leader) = cluster.first_client() else {
            return Err("no kv node clients available".into());
        };

        for idx in 0..self.operations {
            let key = format!("{}-{}", self.key_prefix, idx % self.key_count);
            let response: PutResponse = leader
                .put(&format!("/kv/{key}"), &PutRequest { value: format!("value-{idx}"), expected_version: None })
                .await?;

            if !response.applied {
                return Err(format!("leader rejected write for key {key}").into());
            }

            if let Some(delay) = interval {
                tokio::time::sleep(delay).await;
            }
        }

        Ok(())
    }
}

This workload fetches its cluster handle, writes through one client for a bounded number of operations, controls its rate with a sleep, and returns Err for an unexpected response so the runner stops the run.

The workload is bounded by self.operations, so it terminates on its own; the run duration only decides how long the scenario stays up around it.

Accessing the RunContext

RunContext<E> (testing-framework/core/src/scenario/runtime/context.rs) gives a workload access to:

Notes on the client surface:

• node_clients().snapshot() clones the current client vector so you can iterate across .await points. Use with_clients(|clients| ...) for synchronous reads without the clone.
• extension::<T>() returns a clone of a value registered by a runtime extension factory, for example an ObservationHandle from Continuous Observation.
• ctx.node_control() covers only nodes owned by the outer scenario; in app scenarios node control lives on each cluster’s ClusterHandle instead — see Chaos and Controlled Failure.

Workloads in app scenarios use AppRunContextExt (from testing-framework-app) to reach composed application handles:

use testing_framework_app::AppRunContextExt;

let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;

OpenRaftKvClusterAccessible (examples/openraft_kv/testing/workloads/src/handle_access.rs) uses only require_app to assert that the exposed cluster handle matches the expected topology. See AppHost and with_app for the app layer itself.

See Also

• Expectations and Evaluation — the checks that run after your traffic
• Runtime Extensions — sharing typed values with workloads
• Chaos and Controlled Failure — workloads that restart nodes
• Continuous Observation — polling application state while workloads run

Expectations and Evaluation

Expectations define success conditions. They can capture state before workloads start, check invariants while traffic runs, and evaluate the final state after the run settles.

The Expectation Trait

Expectation<E> lives in testing-framework/core/src/scenario/expectation.rs:

use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};

#[async_trait]
pub trait Expectation<E: Application>: Send + Sync {
    fn name(&self) -> &str;

    fn init(
        &mut self,
        _descriptors: &E::Deployment,
        _run_metrics: &RunMetrics,
    ) -> Result<(), DynError> {
        Ok(())
    }

    async fn start_capture(&mut self, _ctx: &RunContext<E>) -> Result<(), DynError> {
        Ok(())
    }

    /// Optional periodic check used by fail-fast expectation mode.
    async fn check_during_capture(&mut self, _ctx: &RunContext<E>) -> Result<(), DynError> {
        Ok(())
    }

    async fn evaluate(&mut self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

The trait methods are:

• init runs at build() time with the resolved deployment and run metrics; a failure aborts the build.
• start_capture runs once per expectation before any workload starts. Use it to record a baseline (initial counters, starting state). A failure here is ScenarioError::ExpectationCapture and stops the run before traffic begins.
• check_during_capture is a fail-fast hook. The runner calls it on every expectation roughly once per second for the whole workload window (and the cooldown window). The default is a no-op, so existing end-of-run expectations are unaffected. The first check that returns Err aborts the run immediately with ScenarioError::ExpectationFailedDuringCapture. Use it for invariants that must hold throughout the run.
• evaluate checks the final condition after the run settles. It takes &mut self, so it can consume state accumulated during capture.

Registration

Two paths feed the scenario’s expectation list:

1. Explicit: .with_expectation(exp) or .with_expectation_boxed(boxed) on any builder.
2. Workload-attached: when you call .with_workload(w), the builder also collects w.expectations() (see Workloads and Concurrency). The default implementation returns none.

Both end up in the same list and are treated identically at run time.

Workload-attached expectations let a workload register the checks associated with its own traffic. Adding the workload also adds those checks.

Evaluation and Failure Aggregation

flowchart LR
    SC[start_capture]:::sc --> W[Workload window<br/>+ periodic checks]:::sc
    W --> CD[Cooldown + settle]:::sc
    CD --> EV[evaluate all]:::sc
    EV --> R{failures?}
    R -->|no| OK[run passes]
    R -->|yes| AGG[aggregated report]

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

At the end of the run the runner evaluates every registered expectation, even after failures. Each failure is recorded as name: error, and the results are joined into a single ScenarioError::Expectations report:

expectations failed:
kv_converges: kv convergence not reached within 20s for 20 keys
openraft_kv_converges: timed out waiting for observed replicated state convergence ...

This is different from workload failures and capture-check failures, which abort immediately.

Cooldown: with_expectation_cooldown

Workload traffic may need time to settle before evaluation: replication lags, queues drain, and restarted nodes rejoin. The builder exposes:

.with_expectation_cooldown(Duration::from_secs(20))

Verified behavior (runner.rs and definition/validation.rs):

• If you never call it, the cooldown defaults to 10 seconds. (build() also enforces a minimum run duration of 10 seconds.)
• After the workload window, the runner keeps the run alive for the cooldown window, still joining unfinished workloads and still running check_during_capture ticks.
• When the framework owns the node lifecycle and node control was granted (the ClusterApp default), the cooldown window is raised to a minimum of 30 seconds so restarted or freshly deployed nodes stabilize; managed clusters deployed without control skip the floor.
• Before calling evaluate, the runner additionally sleeps a short settle wait derived from the same setting (at least 2 seconds when a cooldown is configured or node control is active) so runtime extensions such as observers catch up.

Set the cooldown to zero only for scenarios without managed nodes where staleness cannot matter.

Worked Example: Convergence Checks

The kvstore example’s KvConverges (examples/kvstore/testing/workloads/src/expectations.rs) is a plain polling expectation. It only implements evaluate and does its own retry loop against the node clients:

use async_trait::async_trait;
use kvstore_runtime_ext::KvEnv;
use testing_framework_app::{AppHostEnv, AppRunContextExt as _};
use testing_framework_core::scenario::{ClusterHandle, DynError, Expectation, RunContext};

#[async_trait]
impl Expectation<AppHostEnv> for KvConverges {
    fn name(&self) -> &str {
        "kv_converges"
    }

    async fn evaluate(&mut self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let cluster = ctx.require_app::<ClusterHandle<KvEnv>>()?;
        let clients = cluster.clients();
        if clients.is_empty() {
            return Err("no kv node clients available".into());
        }

        let deadline = tokio::time::Instant::now() + self.timeout;
        while tokio::time::Instant::now() < deadline {
            if self.is_converged(&clients).await? {
                return Ok(());
            }
            tokio::time::sleep(self.poll_interval).await;
        }

        Err(format!(
            "kv convergence not reached within {:?} for {} keys",
            self.timeout, self.key_count
        )
        .into())
    }
}

The example follows two conventions:

• Poll with a deadline inside evaluate. Eventual consistency is the common case; a one-shot read makes flaky tests.
• Make the error message carry the diagnosis. State what was expected, how long you waited, and (where available) what was last observed.

The openraft_kv variant, OpenRaftKvConverges (examples/openraft_kv/testing/workloads/src/convergence.rs), reads the cluster observer exposed by the app preset instead of querying nodes directly:

async fn evaluate(&mut self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let expected = expected_kv(&self.key_prefix, self.total_writes);
    let observer = ctx.require_app::<ObservationHandle<OpenRaftClusterObserver>>()?;

    wait_for_observed_replication(&observer, &expected, self.timeout).await?;

    Ok(())
}

The observer polls every node in the background, and the expectation waits for a matching snapshot without maintaining its own client polling state. See Continuous Observation for the mechanism.

See Also

• Workloads and Concurrency — the traffic these checks judge
• Continuous Observation — snapshot-based state for expectations
• Runtime Extensions — how extension handles reach evaluate
• Telemetry and External Observability — asserting on Prometheus metrics

The Verb Layer

The verb layer provides optional, domain-specific helpers for recurring test actions. It uses the same scenario builder, workloads, and expectations described in the preceding chapters — a verb chain is sugar over an AppHost scenario.

Two Equivalent Levels

The explicit API names the objects being assembled:

let mut scenario = AppHost::scenario()
    .with_app(ClusterApp::<QueueEnv>::new(QueueTopology::new(5)))
    .with_workload(QueueProduceWorkload::new().operations(400).rate_per_sec(40))
    .with_workload(
        ClusterRestartChaos::<QueueEnv>::new()
            .every_secs(5, 15)
            .excluding_nodes(["node-0"]),
    )
    .with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
    .with_run_duration(Duration::from_secs(120))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The verb API lowers to those same operations:

QueueScenario::nodes(5)
    .produce(400).rate_per_sec(40).done()
    .restart_nodes_randomly().every_secs(5, 15).excluding_nodes(["node-0"]).done()
    .expect_converged(400).within_secs(60)
    .run_secs(120)
    .await?;

Runnable as cargo run -p queue-examples --bin queue_dsl_demo. The explicit API remains available at every point. Use it for one-off workloads, unusual policies, or operations that do not have a domain verb.

How Verbs Map to the Builder

The queue DSL (examples/queue/testing/workloads/src/dsl.rs) shows the pattern. QueueScenario::nodes(n) wraps a fresh AppHost::scenario() builder and remembers the requested node count:

pub struct QueueScenarioBuilder {
    inner: AppHostScenarioBuilder,
    nodes: usize,
}

A verb does not introduce a second runtime. Its sub-builder stores an ordinary workload or expectation and adds it when done() or a terminal method is called:

pub trait QueueDslExt: CoreBuilderAccess<Env = AppHostEnv> + Sized {
    fn produce(self, operations: usize) -> QueueProduceBuilder<Self> {
        QueueProduceBuilder {
            builder: self,
            workload: QueueProduceWorkload::new().operations(operations),
        }
    }
}

impl<B: CoreBuilderAccess<Env = AppHostEnv>> QueueProduceBuilder<B> {
    pub fn done(self) -> B {
        let Self { builder, workload } = self;
        builder.map_core_builder(|inner| inner.with_workload(workload))
    }
}

restart_nodes_randomly() wraps ClusterRestartChaos<QueueEnv> the same way, and expect_converged(n).within_secs(s) lowers to with_expectation(QueueConverges::new(n).timeout(...)).

The finisher does the rest: run_secs(secs) attaches the queue cluster as a ClusterApp, sets the run duration, builds the scenario, and runs it through the backend-neutral deployer:

impl QueueRunExt for QueueScenarioBuilder {
    async fn run_secs(self, secs: u64) -> Result<(), DynError> {
        let Self { inner, nodes } = self;
        let mut scenario = inner
            .with_app(ClusterApp::<QueueEnv>::new(QueueTopology::new(nodes)))
            .with_run_duration(Duration::from_secs(secs))
            .build()
            .map_err(DynError::from)?;

        let runner = AppHostDeployer.deploy(&scenario).await?;
        runner.run(&mut scenario).await?;
        Ok(())
    }
}

Both forms therefore use the same execution, failure aggregation, and teardown. Because the verbs are defined over any CoreBuilderAccess<Env = AppHostEnv> builder, generic and application-specific verbs can extend the same chain.

Verbs and Cluster Control

Some verbs imply a runtime requirement. restart_nodes_randomly() needs node control on the queue cluster’s handle — which ClusterApp requests by default (ClusterControlRequest::Full), so the verb simply adds the chaos workload and relies on the handle. A data-plane verb such as produce(...) needs nothing beyond a client.

Do not hide an unrelated policy choice inside a verb. A verb may configure what its action necessarily needs; retry policy, cleanup policy, backend selection, and other test-wide decisions remain explicit on the cluster app or request.

Designing Application Verbs

Put vocabulary shared by applications in the framework and vocabulary specific to one protocol in that application’s testing crate. A verb should:

1. name an operation in the application’s domain;
2. configure one workload or expectation, or a small fixed combination;
3. expose meaningful options through a short sub-builder;
4. return the underlying builder through done() or a clear terminal method;
5. preserve access to with_workload and with_expectation for uncommon cases.

The queue example keeps random restarts generic — its verb wraps the framework’s ClusterRestartChaos — while produce and expect_converged live with the queue integration. Other applications can then reuse chaos behavior without depending on queue terminology.

See Also

• Workloads and Concurrency and Expectations and Evaluation: the objects verbs add.
• Cluster Control and Observability: the control surface restart verbs rely on.
• Chaos and Controlled Failure: the generic restart workload.

Cluster Control and Observability

Node control and telemetry endpoints are per-cluster request data. Each ClusterRequest states what the test needs from that cluster; the backend provisioner wires the matching runtime surfaces into the returned ClusterHandle. Operations a cluster does not have fail with a clear error at the call site instead of somewhere inside a workload.

Control as Request Data

ClusterControlRequest (testing-framework/core/src/scenario/provisioning.rs) has two values:

A raw ClusterRequest defaults to None. ClusterApp defaults to Full, because a cluster deployed as the system under test usually wants restarts available; opt out with .with_control(ClusterControlRequest::None):

let scenario = AppHost::scenario()
    .with_app(
        ClusterApp::<QueueEnv>::new(QueueTopology::new(3))
            .with_control(ClusterControlRequest::None),   // clients only
    )
    .build()?;

What Full buys depends on the backend: local grants the complete surface including StartNodeOptions, compose grants restart (plus stop for attached clusters), and k8s grants start/stop/restart by scaling per-node deployments. The Backend Capability Matrix records the details.

Every handle also reports a ClusterControlProfile describing who owns the lifecycle: FrameworkManaged (provisioned eagerly, torn down by the framework), ManualControlled (provisioned on demand, your code starts nodes), ExistingClusterAttached, or ExternalUncontrolled. framework_owns_lifecycle() is true only for FrameworkManaged.

Node Control Without ManualCluster

Restarting nodes from a declarative workload does not require ManualCluster. The handle carries the control surface:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<ClusterHandle<QueueEnv>>()?;

    cluster.restart_node("node-1").await?;
    cluster.wait_node_ready("node-1").await?;
    Ok(())
}

When control was not requested (or the backend cannot provide an operation), the call returns an error such as "cluster node control is not available" — explicit partial support at run time. Chaos and Controlled Failure shows full restart scenarios built this way, including the reusable ClusterRestartChaos workload. ManualCluster remains the imperative API for tests that control the entire node lifecycle themselves; see ManualCluster: Imperative Node Control.

The Handle’s Control Surface

ClusterHandle<E> forwards these operations to the backend’s NodeControlHandle (testing-framework/core/src/scenario/control.rs):

StartedNode<E> is a plain pair: the node name and a fresh E::NodeClient.

wait_network_ready() is the matching cluster-wide wait, backed by the provisioner’s ClusterWaitHandle. Inside workloads, prefer waiting on observed application state instead.

Start Modes

ClusterStartMode selects when managed nodes start:

• Eager (default): the provisioner starts and readiness-gates every node before the handle is returned.
• OnDemand: the cluster is prepared but no node runs; the handle’s profile is ManualControlled and your code calls start_node explicitly. Supported by the local and k8s provisioners; the compose provisioner rejects it.

ClusterApp::<QueueEnv>::new(QueueTopology::new(3))
    .with_start_mode(ClusterStartMode::OnDemand)

StartNodeOptions

StartNodeOptions<E> customizes a dynamic start or restart. Overview (full treatment in Part IV: Ports, Peers, Node Config, and Readiness and Persistence, Snapshots, and Recovery Testing):

The local backend honors all of them; k8s rejects persist/snapshot dirs and routes config overrides through cfgsync where the environment supports it (Kubernetes Backend).

Observability as Request Data

ObservabilityInputs (testing-framework/core/src/scenario/observability.rs) carries optional telemetry endpoints:

Set it per cluster:

let observability = ObservabilityInputs {
    metrics_query_url: Some(Url::parse("http://127.0.0.1:19091")?),
    ..ObservabilityInputs::default()
};

let app = ClusterApp::<MetricsCounterEnv>::new(MetricsCounterTopology::new(3))
    .with_observability(observability);

The compose and k8s provisioners resolve the effective inputs by reading the LOGOS_BLOCKCHAIN_METRICS_QUERY_URL, LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL, and LOGOS_BLOCKCHAIN_GRAFANA_URL environment variables and applying the request’s values as overrides (request wins). The OTLP ingest URL flows into node config preparation; the resolved inputs are stored on the handle. The local provisioner does not populate observability, so local handles report empty metrics.

Expectations read metrics through the handle:

let cluster = ctx.require_app::<ClusterHandle<MetricsCounterEnv>>()?;
let metrics = cluster.metrics()?;             // Prometheus-backed when configured
let total = metrics.counter_value("sum(metrics_counter_increments_total)")?;

metrics() builds a Prometheus-backed Metrics handle from metrics_query_url, or an empty handle when none is configured (is_configured() distinguishes the two). Details, including what telemetry is and is not, are in Telemetry and External Observability.

See Also

• Chaos and Controlled Failure — node control from workloads
• ManualCluster: Imperative Node Control — the imperative alternative
• Telemetry and External Observability — observability inputs in use
• Backend Capability Matrix — backend support by feature

Chaos and Controlled Failure

Chaos scenarios deliberately stop or restart nodes and then check recovery. Ordinary workloads perform these operations through the deployed cluster’s ClusterHandle.

The Shape of a Chaos Scenario

A chaos test is three ordinary pieces wired together:

1. A cluster deployed with node control — ClusterApp requests ClusterControlRequest::Full by default (Cluster Control and Observability).
2. A workload that drives traffic, disrupts a node via the handle, waits for recovery, and drives traffic again — or the reusable ClusterRestartChaos workload for random restarts.
3. An expectation that verifies the end state converged despite the disruption.

flowchart LR
    T[Drive traffic]:::sc --> D[Disrupt<br/>restart node]:::sc
    D --> W[Wait for recovery<br/>observed state]:::sc
    W --> T2[Drive traffic again]:::sc
    T2 --> V[Expectation:<br/>state converged]:::sc

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

No ManualCluster is involved: the provisioner wires a node-control surface into the handle because the cluster request asked for it, and this works on every backend whose provisioner grants control.

The Built-In Restart Workload

ClusterRestartChaos<E> (testing-framework-app) randomly restarts nodes of one deployed cluster application. It resolves the cluster through its exposed ClusterHandle<E> and drives restarts through that handle:

let mut scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<KvEnv>::new(KvTopology::new(3)),
        K8sClusterProvisioner,
    )
    .with_run_duration(Duration::from_secs(90))
    .with_workload(
        ClusterRestartChaos::<KvEnv>::new()
            .every_secs(10, 20)        // random delay window between restarts
            .cooldown_secs(60)         // per-node minimum time between restarts
            .excluding_nodes(["node-0"]),
    )
    .with_workload(
        KvWriteWorkload::new()
            .operations(300)
            .key_count(20)
            .rate_per_sec(10)
            .key_prefix("kv-chaos"),
    )
    .with_expectation(KvConverges::new("kv-chaos", 20).timeout(Duration::from_secs(60)))
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

This is the shipped kvstore_k8s_restart_chaos binary, verifying managed k8s node control end to end; swap the provisioner for LocalClusterProvisioner (or plain with_app) and the same scenario runs on local processes.

The knobs:

Targets are the cluster’s node-<index> names; a cluster with one node (or none after exclusions) fails fast with “no eligible targets”. The workload runs until the scenario’s run duration elapses.

The queue verb DSL exposes the same workload as a verb: .restart_nodes_randomly().every_secs(5, 15).done() (The Verb Layer).

Worked Example: OpenRaft Leader Failover

The openraft_kv failover test bootstraps a three-node Raft cluster, expands it to three voters, writes a batch, restarts the leader, then writes a second batch through the node elected next. Its workload is in examples/openraft_kv/testing/workloads/src/failover.rs:

#[async_trait]
impl Workload<AppHostEnv> for OpenRaftKvFailoverWorkload {
    fn name(&self) -> &str {
        "openraft_kv_failover_workload"
    }

    async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let cluster = ctx.require_app::<ClusterHandle<OpenRaftKvEnv>>()?;
        let clients = cluster.clients();
        let observer = ctx.require_app::<ObservationHandle<OpenRaftClusterObserver>>()?;

        ensure_cluster_size(&clients, 3)?;
        self.bootstrap_cluster(&clients).await?;

        let initial_leader = wait_for_observed_leader(&observer, self.timeout, None).await?;
        let membership = OpenRaftMembership::discover(&clients).await?;

        self.promote_cluster(&observer, &clients, initial_leader, &membership).await?;
        self.write_initial_batch(&clients, initial_leader).await?;

        let new_leader = self
            .restart_leader_and_wait_for_failover(&cluster, &observer, initial_leader)
            .await?;
        self.write_second_batch(&clients, new_leader).await?;

        Ok(())
    }
}

The disruption itself is a few lines:

cluster.restart_node(&leader_name).await?;

let new_leader = wait_for_observed_leader(observer, self.timeout, Some(leader_id)).await?;

If the cluster was deployed without control, restart_node returns a clear “cluster node control is not available” error.

Assembling the Scenario

build_failover_scenario (examples/openraft_kv/examples/src/lib.rs) puts workload, expectation, and cluster together, parameterized by the backend provisioner:

pub fn build_failover_scenario<P>(
    run_duration: Duration,
    workload_timeout: Duration,
    provisioner: P,
) -> anyhow::Result<Scenario<AppHostEnv>>
where
    P: ClusterProvisioner<OpenRaftKvEnv>,
{
    Ok(AppHost::scenario()
        .with_app_using(OpenRaftKvClusterApp::nodes(3), provisioner)
        .with_run_duration(run_duration)
        .with_workload(OpenRaftKvClusterAccessible::new(3))
        .with_workload(
            OpenRaftKvFailoverWorkload::new()
                .first_batch(INITIAL_WRITE_BATCH)
                .second_batch(SECOND_WRITE_BATCH)
                .timeout(workload_timeout)
                .key_prefix(RAFT_KEY_PREFIX),
        )
        .with_expectation(
            OpenRaftKvConverges::new(TOTAL_WRITES)
                .timeout(run_duration)
                .key_prefix(RAFT_KEY_PREFIX),
        )
        .build()?)
}

Run it locally or on compose:

cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover
cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover

The openraft_kv_k8s_failover bin executes the same failover flow on Kubernetes, but uses ManualCluster imperatively (start_node per node, restart_node, wait_network_ready). See ManualCluster: Imperative Node Control.

Patterns

Restart-and-verify. The minimal chaos loop: write known data, restart_node, wait for readiness, verify the data survived. Restarts reuse the node’s existing working directory, so on-disk state survives them by default; use with_snapshot_dir to seed a restore from saved state; details are in Persistence, Snapshots, and Recovery Testing.

Leader failover. Restart the node that currently holds a distinguished role. The failover workload discovers the leader from observed cluster state instead of assuming a node index. Passing the old identity to the wait (different_from: Some(leader_id) above) verifies that leadership changed rather than accepting the old leader after it restarts.

Readiness waits after disruption. Wait before sending traffic after a restart. The example waits on observed application state (an agreed leader across all nodes) via an observation handle, which checks more than an HTTP readiness probe. In imperative flows, ManualCluster::wait_network_ready() covers transport-level readiness (the k8s bin calls it right after restart_node); inside a declarative workload, wait on observed state or cluster.wait_node_ready(name).

Pair chaos with continuous observation. A background observer polls every node through the disruption, so waits read stored snapshots and can report the last observation (timed out waiting for observed leader agreement ...; last observation: node=0 leader=None ...). A workload can also poll clients directly, but must then track the polling state itself.

Give state time to settle. Set with_expectation_cooldown(...) on the builder when post-chaos state needs a settle window between the workload phase and evaluation; see Expectations and Evaluation.

See Also

• Cluster Control and Observability — control requests and StartNodeOptions
• Continuous Observation — the observer used for recovery waits
• Persistence, Snapshots, and Recovery Testing — restart with retained state
• ManualCluster: Imperative Node Control — the imperative variant

Runtime Extensions

Runtime extensions are typed values prepared once per run (after nodes exist, before workloads start) and handed to workloads and expectations through the RunContext. The app layer and the observation runtime are built on them.

The Mechanism

The implementation is in testing-framework/core/src/scenario/runtime/extensions.rs and has three parts:

1. A factory registered on the builder. RuntimeExtensionFactory<E> is called by the deployer during preparation, when the deployment is resolved and node clients are available:

#[async_trait]
pub trait RuntimeExtensionFactory<E: Application>: Send + Sync {
    async fn prepare(
        &self,
        deployment: &E::Deployment,
        node_clients: NodeClients<E>,
    ) -> Result<PreparedRuntimeExtension, DynError>;
}

Register it with .with_runtime_extension_factory(Box::new(factory)). Factories run in registration order; any prepare error aborts the deployment.

The factory runs after the deployment is resolved and node clients exist. It prepares the value once so every workload and expectation can share it instead of rebuilding the same clients or polling loops.

2. A prepared value, optionally with cleanup. PreparedRuntimeExtension wraps one value of any Clone + Send + Sync + 'static type, with three constructors:

Cleanup guards are collected into the run’s cleanup chain and execute at teardown in reverse registration order; see Handle Ownership and Teardown for how the app layer separates those guards from handle access.

3. Typed retrieval from the context. The prepared values land in a type-indexed store inside RunContext:

// Somewhere in a workload or expectation:
let handle: MyHandle = ctx.require_extension::<MyHandle>()?;
// or, tolerating absence:
let maybe: Option<MyHandle> = ctx.extension::<MyHandle>();

extension::<T>() returns a clone of the stored value. Extension values should therefore be cheap to clone, typically by wrapping shared state in an Arc.

One Value Per Type

The store is keyed by TypeId. Registering two extensions that prepare the same type is a hard error at prepare time:

duplicate runtime extension type registered: <type name>

Because ctx.extension::<T>() returns one value by type, each type may be registered only once. To register several values with the same underlying shape, wrap them in distinct newtypes or, in the app layer, use named handles instead (see AppDeployment and DeployContext).

The app layer is the exception that proves the rule: its AppRuntime extension is registered as mergeable, so several with_app(...) calls combine into one handle registry instead of colliding — duplicate detection moves to the handle level (type plus name). See AppHost and with_app.

Writing a Factory

A minimal factory that shares a client wrapper with all workloads:

use async_trait::async_trait;
use testing_framework_core::scenario::{
    DynError, NodeClients, PreparedRuntimeExtension, RuntimeExtensionFactory,
};

#[derive(Clone)]
struct FrontDoor(MyNodeClient);

struct FrontDoorFactory;

#[async_trait]
impl RuntimeExtensionFactory<MyEnv> for FrontDoorFactory {
    async fn prepare(
        &self,
        _deployment: &<MyEnv as Application>::Deployment,
        node_clients: NodeClients<MyEnv>,
    ) -> Result<PreparedRuntimeExtension, DynError> {
        let client = node_clients
            .snapshot()
            .first()
            .cloned()
            .ok_or("no nodes available")?;

        Ok(PreparedRuntimeExtension::new(FrontDoor(client)))
    }
}

// Registration:
let builder = builder.with_runtime_extension_factory(Box::new(FrontDoorFactory));

For an extension backed by a polling loop, spawn the task in prepare and return from_task(handle, join_handle). The runner aborts the task when the run tears down, so the loop cannot outlive the cluster. The observation runtime works this way. Inside an AppDeployment, the equivalent pattern is spawning the task in deploy and registering a guard with ctx.defer_cleanup(...), as the pubsub example’s PubSubStackApp does for its topic feed (examples/pubsub/testing/integration/src/feed.rs).

What Is Built on This

The following layers use runtime extension factories:

So when a workload calls ctx.require_app::<ClusterHandle<KvEnv>>() or ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>(), it is walking the same type-indexed store described above.

• App layer: AppHost and with_app
• Observation runtime: Continuous Observation

See Also

• Workloads and Concurrency — consuming extensions from workloads
• Continuous Observation — an extension backed by a polling task
• Handle Ownership and Teardown — cleanup ordering in depth

Continuous Observation

The observation runtime polls application state in the background and stores snapshots, histories, and event streams for workloads and expectations. It provides typed state inside the test process rather than external telemetry.

Shared Polling Runtime

Chaos and convergence tests repeatedly query state such as the current leader, whether every node has seen a key, or what changed after a restart. The observation runtime (testing-framework/core/src/observation/) runs one background polling task with shared error and staleness tracking. Workloads and expectations read the stored state.

Telemetry exports metrics, logs, and traces to external endpoints. Observation instead keeps typed application state inside the test and makes it synchronously queryable during the run.

The Observer Trait

An application defines how to poll and interpret its state; the runtime schedules the polling:

#[async_trait]
pub trait Observer: Send + Sync + 'static {
    type Source: Clone + Send + Sync + 'static;   // app-owned source handle
    type State: Send + Sync + 'static;            // retained materialized state
    type Snapshot: Clone + Send + Sync + 'static; // current view
    type Event: Clone + Send + Sync + 'static;    // delta emitted per cycle

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError>;

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError>;

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot;
}

Each cycle the runtime refreshes the source set, calls poll to advance State and collect delta Events, then derives a Snapshot from the state. ObservedSource<S> is just a name plus the app-owned source value (ObservedSource::new(name, source)), typically a node client.

Sources are re-queried every cycle through SourceProvider<S>:

#[async_trait]
pub trait SourceProvider<S>: Send + Sync + 'static {
    async fn sources(&self) -> Result<Vec<ObservedSource<S>>, DynError>;
}

StaticSourceProvider::new(sources) covers the common fixed-cluster case. A custom provider makes sources dynamic: it can return a different set each cycle, which lets observation stay correct across node restarts. SourceProviderFactory<E, S> builds the provider once node clients exist; any closure Fn(&E::Deployment, NodeClients<E>) -> Result<BoxedSourceProvider<S>, DynError> qualifies.

Plugging Into a Scenario

ObservationExtensionFactory<E, O> is a runtime extension factory: at prepare time it builds the source provider, starts the runtime, and stores the read handle in the RunContext (background task registered for abort-on-teardown via PreparedRuntimeExtension::from_task). The builder has convenience methods for it (CoreBuilderExt):

// Clonable observer:
builder.with_observer(MyObserver, my_source_provider_fn, ObservationConfig::default())
// Observer built lazily per run:
builder.with_observer_factory(|| MyObserver::new(), my_source_provider_fn, config)

ObservationConfig has two fields: interval (time between cycles, default 1 s, must be non-zero) and history_limit (retained non-empty event batches, default 64).

Outside scenarios, for example around a ManualCluster, start it directly: ObservationRuntime::start(provider, observer, config), then handle(), into_parts() (handle + JoinHandle), or abort(). Dropping the runtime aborts the task.

Reading: the ObservationHandle

Workloads and expectations retrieve the handle by type and read four things:

Snapshots vs batches vs events: a snapshot is the whole current view (cycle, observed_at, source_count, value); an event is one delta discovered during a cycle; a batch groups the events of one cycle. Cycles that produce no events produce no batch. history() and subscribe() only ever see non-empty batches, while latest_snapshot() is refreshed on every successful cycle.

Freshness and failures. On a failed cycle, the runtime records an ObservationFailure (with stage: SourceRefresh if source discovery failed, stage: Poll if the observer failed) and retains the last successful snapshot. The next successful cycle clears last_error. To check staleness, compare snapshot.cycle or observed_at across reads, and inspect last_error() when a wait times out; it usually names the source that stopped answering.

Worked Example: the OpenRaft Cluster Observer

examples/openraft_kv/testing/integration/src/observation.rs observes a Raft cluster. State and snapshot are the same type (the latest per-node states plus any per-source failures), and no delta events are emitted (Event = ()):

#[derive(Clone, Debug, Default)]
pub struct OpenRaftClusterObserver;

#[async_trait]
impl Observer for OpenRaftClusterObserver {
    type Source = OpenRaftKvClient;
    type State = OpenRaftClusterSnapshot;
    type Snapshot = OpenRaftClusterSnapshot;
    type Event = ();

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError> {
        Ok(capture_cluster_snapshot(sources).await)
    }

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError> {
        *state = capture_cluster_snapshot(sources).await;
        Ok(Vec::new())
    }

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot {
        state.clone()
    }
}

capture_cluster_snapshot queries each source’s /state endpoint and records per-node errors as OpenRaftSourceFailure values instead of failing the cycle. A node restarting therefore appears as a named failure inside the snapshot. The snapshot type provides agreed_leader(different_from), all_voters_match(...), all_kv_match(...), and summary() for timeout messages.

Two source providers accompany it: OpenRaftNodeClientsSourceProvider, which re-reads the cluster’s shared NodeClients every cycle so observation follows restarts inside a scenario, and OpenRaftManualClusterSourceProvider, which re-resolves clients from a ManualCluster for imperative runs.

The example’s OpenRaftKvClusterApp preset (examples/openraft_kv/testing/integration/src/app.rs) deploys the cluster, starts ObservationRuntime::start(provider, OpenRaftClusterObserver, config) inside its deploy, registers the polling task with ctx.defer_cleanup(...), and exposes the ObservationHandle as an app handle next to the ClusterHandle.

The failover scenario waits on this observed state:

let observer = ctx.require_app::<ObservationHandle<OpenRaftClusterObserver>>()?;
let leader = wait_for_observed_leader(&observer, timeout, None).await?;

External example: logos-blockchain’s BlockFeed is an adopter-side analog of this pattern: an observer in its own repository materializes block records, per-node head snapshots, and transaction statistics, using the same Observer/ObservationHandle mechanism.

See Also

• Runtime Extensions — the mechanism observation plugs into
• Chaos and Controlled Failure — observation-driven recovery waits
• Expectations and Evaluation — snapshot-based verdicts
• Telemetry and External Observability — the external counterpart

Telemetry and External Observability

Telemetry connects a scenario to external observability infrastructure such as Prometheus, an OTLP collector, and Grafana. It supports PromQL queries and external dashboards. For typed application state inside a test, use the observation runtime.

Observation and Telemetry

Telemetry endpoints are optional on every backend. Without telemetry configuration, the scenario still runs and the cluster’s metrics() handle has no Prometheus backend.

Declaring Endpoints on the Cluster

ObservabilityInputs (testing-framework/core/src/scenario/observability.rs) carries three optional URLs:

Endpoints travel per cluster on the ClusterRequest. Set them with ClusterApp::with_observability (or ClusterRequest::with_observability inside a custom AppDeployment):

use testing_framework_core::scenario::ObservabilityInputs;

let inputs = ObservabilityInputs {
    metrics_query_url: Some("http://127.0.0.1:9090".parse()?),
    metrics_otlp_ingest_url: Some("http://127.0.0.1:4318".parse()?),
    grafana_url: None,
};

let scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<MetricsCounterEnv>::new(topology).with_observability(inputs),
        ComposeProvisioner::default(),
    )
    .build()?;

Environment Plus Request

Backend provisioners resolve effective inputs by merging two sources: environment values form the base, and any endpoint set on the cluster request overrides the corresponding environment value:

let inputs = ObservabilityInputs::from_env()?
    .with_overrides(request.observability().clone());

This allows the environment to supply infrastructure-specific endpoints while the scenario can override individual URLs per cluster.

What from_env reads. Verified against the source, it reads exactly three environment variables, each parsed as a URL (empty or unset values are skipped; an unparsable value is an error):

ObservabilityInputs also offers with_overrides(other) (field-wise, Some wins) and telemetry_handle(), which builds the Metrics value exposed through the ClusterHandle: Metrics::from_prometheus(url) when metrics_query_url is set, Metrics::empty() otherwise.

Backend support today: the compose and k8s provisioners resolve env + request as above and wire the OTLP ingest URL into node configuration. The local provisioner does not wire telemetry endpoints into node configuration. See the Capability Matrix.

Querying Metrics in a Run

ClusterHandle::metrics() returns the Metrics handle for that cluster’s endpoints. Backed by Prometheus it evaluates instant queries:

let cluster = ctx.require_app::<ClusterHandle<MetricsCounterEnv>>()?;
let telemetry = cluster.metrics()?;
let values = telemetry.instant_values("up")?;          // all sample values
let total = telemetry.counter_value("requests_total")?; // summed counter

Without a configured metrics_query_url, is_configured() is false and query calls return a MetricsError (“prometheus endpoint unavailable”). Expectations that assert on metrics therefore require a configured telemetry endpoint.

Telemetry queries depend on scrape intervals, exporter lag, and external infrastructure. Observation polls application state from the test process and reports failures by source. Correctness checks can use observation when they require current typed state; performance checks and post-run analysis can use telemetry.

A Local Stack for Development

To use a local Prometheus, OTLP collector, and Grafana stack, export the three environment variables above or set the URLs on the cluster request. The same scenario binary can then run with or without a metrics backend.

See Also

• Continuous Observation — test-visible state, the in-process counterpart
• Capability Matrix — per-backend telemetry support
• Environment Variables — the full audited env var list

Part IV — Uniform Clusters and Configuration

This part shows how to put your own node behind the framework and control how clusters are configured and driven.

It covers the uniform-cluster entry pattern in depth: implementing Application, describing topologies, generating per-node configuration, and driving nodes imperatively outside the scenario runtime.

• Implementing Application — the environment contract for your node
• Topology and Deployment Plans — describing cluster shape
• Ports, Peers, Node Config, and Readiness — per-node configuration mechanics
• Static Artifacts and cfgsync — typed app config to per-node artifacts to backend rendering
• Seeds and Reproducibility — deterministic deployments
• ManualCluster: Imperative Node Control — direct node lifecycle control
• Persistence, Snapshots, and Recovery Testing — state across restarts

Implementing Application

This chapter shows how to put your own node binary behind the framework so the cluster provisioners can launch it as a uniform cluster.

The Application Trait

Every environment starts with Application (testing-framework/core/src/env.rs). It bundles the backend-agnostic types the scenario engine needs:

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str;
}

The associated types and methods are:

Workloads and expectations use only these types, so the same scenario code can run against local processes, Compose containers, and Kubernetes services. Each backend still requires its corresponding deployment integration.

Application does not specify how nodes run. Each deployer adds a backend-specific integration trait.

Local Integration: Two Paths

The local deployer (testing-framework/deployers/local/src/env/mod.rs) offers two traits.

LocalBinaryApp covers apps that launch one binary per node, write one config file per node, and expose one HTTP API port. You implement five methods; a blanket implementation supplies LocalDeployerEnv:

Optional overrides: initial_local_port_names() (extra named ports reserved per node), readiness_endpoint_path(), readiness_probe() (HTTP GET or plain TCP), and wait_readiness_stable(nodes) for app-specific stabilization after the port probe succeeds.

LocalDeployerEnv exposes the deployer-facing hooks directly: build_node_config_from_template, build_initial_node_configs, build_launch_spec, node_endpoints, node_client, node_peer_port, local_process_spec_for_node (per-node binary selection for mixed-version clusters), and initial_persist_dir / initial_snapshot_dir (see Persistence). Implement it directly when LocalBinaryApp does not cover the application’s launch requirements.

graph LR
    A[Application] --> B[LocalBinaryApp]
    B -- blanket impl --> C[LocalDeployerEnv]
    C --> D["LocalClusterProvisioner"]

Worked Example: kvstore

The kvstore integration lives in examples/kvstore/testing/integration/src/. The environment type is an empty struct:

pub struct KvEnv;

#[async_trait]
impl Application for KvEnv {
    type Deployment = KvTopology;        // = ClusterTopology
    type NodeClient = KvHttpClient;
    type NodeConfig = KvNodeConfig;

    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError> {
        Ok(KvHttpClient::new(access.api_base_url()?))
    }

    fn node_readiness_path() -> &'static str {
        "/health/ready"
    }
}

Client construction. build_node_client receives NodeAccess, a host plus API port (and optional testing/named ports) discovered by the deployer, and wraps its base URL in the app’s HTTP client. The same function serves every backend: local processes, Compose containers, and K8s services all resolve to a NodeAccess.

Readiness path. node_readiness_path returns /health/ready. Deployers append it to http://<host>:<api-port> and poll until the node answers 2xx. See Ports, Peers, Node Config, and Readiness for the probe implementation.

The local side (local_env.rs) implements LocalBinaryApp:

impl LocalBinaryApp for KvEnv {
    fn initial_node_name_prefix() -> &'static str {
        "kv-node"
    }

    fn build_local_node_config_with_peers(
        _topology: &Self::Deployment,
        index: usize,
        ports: &LocalNodePorts,
        peers: &[LocalPeerNode],
        _peer_ports_by_name: &HashMap<String, u16>,
        _options: &StartNodeOptions<Self>,
        _template_config: Option<&KvNodeConfig>,
    ) -> Result<KvNodeConfig, DynError> {
        build_local_cluster_node_config::<Self>(index, ports, peers)
    }

    fn local_process_spec() -> LocalProcessSpec {
        LocalProcessSpec::new("KVSTORE_NODE_BIN")
            .with_binary_provider(kvstore_binary_provider())
            .with_rust_log("kvstore_node=info")
    }

    fn render_local_config(config: &KvNodeConfig) -> Result<Vec<u8>, DynError> {
        yaml_node_config(config)
    }

    fn http_api_port(config: &KvNodeConfig) -> u16 {
        config.http_port
    }
}

Config generation. kvstore delegates to build_local_cluster_node_config::<Self>, which works because KvEnv also implements ClusterNodeConfigApplication (app.rs): a backend-neutral hook that builds a NodeConfig from a ClusterNodeView (own index, host, ports) and ClusterPeerView list. Implementing that one trait gives kvstore local config generation and the static-artifact path used by Compose/K8s; see Static Artifacts and cfgsync.

Binary provider. kvstore_binary_provider() returns a FallbackBinaryProvider chain: first EnvBinaryProvider::new("KVSTORE_NODE_BIN") (use a prebuilt binary if the env var is set), then BuildBinaryProvider running cargo build -p kvstore-node --bin kvstore-node in the workspace root. This is why kvstore examples need no manual setup. Providers are covered in Binary Providers.

Launch and config rendering. At spawn time the framework renders the config with render_local_config, writes it as config.yaml into the node’s working directory, and launches <binary> --config config.yaml with the spec’s env vars. LocalProcessSpec supports different file names, positional config arguments, and extra args (see node-config.md).

With the environment in place, running kvstore on any backend is a matter of which provisioner the scenario passes:

// local (default)
AppHost::scenario().with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))
// compose
    .with_app_using(ClusterApp::<KvEnv>::new(topology), ComposeProvisioner::default())
// k8s
    .with_app_using(ClusterApp::<KvEnv>::new(topology), K8sClusterProvisioner)

Other Backends

The same KvEnv gains container support in two short files:

• compose_env.rs implements ComposeBinaryApp: a BinaryConfigNodeSpec naming the in-container binary path, config path, and exposed ports. See Compose Deployer.
• k8s_env.rs implements K8sBinaryApp: a BinaryConfigK8sSpec with release name, node-name prefix, binary and config paths, and service ports. See Kubernetes Deployer.

Both backends deliver generated configs through cfgsync rather than the local filesystem (Static Artifacts and cfgsync).

Implement traits only for the backends you use. An Application implementation plus LocalBinaryApp is sufficient for local scenarios.

Topology and Deployment Plans

This chapter explains how a scenario describes cluster shape and how a deployment provider turns that description into the concrete deployment the runner uses.

DeploymentDescriptor

The core contract is defined in testing-framework/core/src/topology/mod.rs:

pub trait DeploymentDescriptor: Send + Sync {
    fn node_count(&self) -> usize;
}

Every Application::Deployment implements it. The scenario engine itself only needs the node count; everything richer (per-node configs, ids, network layout) belongs to the app’s own deployment type and to the deployer that interprets it.

Built-in Topology Types

The topology module ships a few concrete building blocks. Verify against testing-framework/core/src/topology/:

Every example app that runs as a uniform cluster aliases ClusterTopology:

pub type KvTopology = testing_framework_core::topology::ClusterTopology;

let topology = KvTopology::new(3); // 3 nodes, indices 0..3

DeploymentPlan and NodePlan implement DeploymentDescriptor too, for apps whose deployment must carry a prebuilt per-node config (plans[i].general) instead of deriving configs at spawn time. TopologyShapeBuilder and DeploymentPlan are available building blocks; the in-repo example apps currently build on ClusterTopology directly.

Deployment Providers

A scenario does not have to hold a finished deployment. It holds a provider:

pub trait DeploymentProvider<D>: Send + Sync
where
    D: DeploymentDescriptor,
{
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

FixedDeploymentProvider<D> wraps a concrete deployment and clones it on every build, ignoring the seed. A custom provider can generate the deployment lazily: sized from the environment, randomized from the seed, or derived from an external inventory.

Feeding the Builder

ScenarioBuilder<E> accepts a deployment in three ways (core/src/scenario/definition/builder.rs):

Resolution happens once, inside build(): the builder calls provider.build(seed) and bakes the resulting deployment into the Scenario. Provisioners and workloads then see a fixed descriptor for the rest of the run.

graph LR
    P[DeploymentProvider] -- "build(seed)" --> D[E::Deployment]
    S[with_deployment_seed] -. optional .-> P
    D --> SC["Scenario<E>"]
    SC --> R[Provisioner / Runner]
    D:::cl
    SC:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

In an app scenario the deployment usually enters as the ClusterApp’s topology instead:

AppHost::scenario().with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)))

map_deployment_provider and with_deployment_provider exist on ScenarioBuilder<E> and on the shared CoreBuilderExt used by app-specific builder extensions.

What the Deployment Does Downstream

• The local provisioner reads node_count() and asks the environment to reserve ports and build one config per index; see Ports, Peers, Node Config, and Readiness.
• The container backends iterate indices to produce per-node static artifacts delivered through cfgsync; see Static Artifacts and cfgsync.
• ManualCluster treats the deployment as capacity: nodes are started on demand against the descriptor. See ManualCluster.

Ports, Peers, Node Config, and Readiness

This chapter describes how the local deployer allocates ports, wires peers, materializes per-node configs, and decides when a cluster is ready.

Port Allocation

All local ports come from the OS. preallocate_ports (in testing-framework/deployers/local/src/env/helpers.rs) binds 127.0.0.1:0, records the assigned port, and releases the listener. reserve_local_node_ports(count, names, label) does this for every node up front and returns one LocalNodePorts per node:

Named ports exist for apps that need more than one listener per node. Declare them via LocalBinaryApp::initial_local_port_names() (or LocalDeployerEnv::local_port_names()); the deployer reserves one port per name per node.

Ports are reserved by bind-and-release, so they are free at reservation time but are not deterministic across runs. See Seeds and Reproducibility.

Peer Wiring

Peer wiring is why up-front port reservation matters: because every node’s ports are reserved before any config is built, each node’s config can reference the real addresses of all its peers before a single process starts.

For each node index the deployer builds peer views of every other node:

• LocalPeerNode: index(), network_port(), http_address() / authority() (127.0.0.1:<port>).
• build_local_peer_nodes(peer_ports, self_index): full peer views, skipping self.
• build_indexed_http_peers(node_count, self_index, peer_ports, build_peer): map peers through your own constructor.

These flow into the app’s config hook together with the node’s own ports:

fn build_local_node_config_with_peers(
    topology: &Self::Deployment,
    index: usize,
    ports: &LocalNodePorts,
    peers: &[LocalPeerNode],
    peer_ports_by_name: &HashMap<String, u16>,
    options: &StartNodeOptions<Self>,
    template_config: Option<&Self::NodeConfig>,
) -> Result<Self::NodeConfig, DynError>;

For initial cluster startup the deployer calls this once per index with every other node as a peer (a full mesh view); the layout your nodes actually form is up to the config your app builds from those views. LocalBuildContext carries the same fields when you customize build_initial_node_configs on the full LocalDeployerEnv path. Apps that implement ClusterNodeConfigApplication can delegate the whole hook to build_local_cluster_node_config::<E>(index, ports, peers), the same abstraction the container backends reuse (see cfgsync).

Config Templates: LocalProcessSpec

LocalProcessSpec describes how one rendered config becomes a running process:

Rendering helpers: yaml_node_config (serialize to YAML bytes), text_node_config (already-rendered text), yaml_config_launch_spec / text_config_launch_spec / default_yaml_launch_spec (build a full LaunchSpec in one call). The resulting LaunchSpec lists the binary, the files to materialize, args, and env; the deployer writes the files into the node’s working directory and spawns the process there.

StartNodeOptions: Overrides at Start Time

Dynamically started nodes (node-control workloads and ManualCluster) accept StartNodeOptions<E>; the full field table lives in the ManualCluster chapter. The two config-shaping fields deserve care:

config_override replaces the complete generated config. config_patch (set via create_patch(|config| ...)) transforms the generated config, retaining framework-assigned ports and peers unless the callback changes them. A full override must provide every required port itself.

Where they are honored differs by path:

• Local dynamic starts (NodeManager::start_node_with): the framework builds the config through the env hooks (which receive the full options) and then applies config_patch itself. config_override and peers are visible to your build_local_node_config_with_peers implementation but are not interpreted centrally by the local path.
• Static-artifact path (used by the container backends through StaticNodeConfigProvider::build_node_artifacts_for_options, core/src/scenario/config.rs): the framework interprets everything: PeerSelection picks the peer set, then config_override replaces, then config_patch transforms, and the result is served as an override artifact.

PeerSelection variants (core/src/scenario/control.rs):

Readiness

Per-node probe. The local deployer probes each node’s API port using LocalDeployerEnv::readiness_probe():

• LocalReadinessProbe::HttpGet { path } (default): GET http://127.0.0.1:<api-port><path> until it returns 2xx. The path defaults to Application::node_readiness_path() ("/" unless overridden; kvstore uses /health/ready).
• LocalReadinessProbe::Tcp: the port merely accepts TCP connections. Use for nodes without an HTTP surface.

Cluster requirement. HttpReadinessRequirement (core/src/scenario/runtime/readiness.rs) decides how many nodes must pass:

Set it as part of the cluster’s DeploymentPolicy (ClusterApp::with_policy); see Readiness, Retry, and Artifact Preservation.

Waiting imperatively. ManualCluster (and ClusterHandle) expose:

• wait_network_ready(): polls every started node’s API port with AllNodesReady.
• wait_node_ready(name): polls one node, honoring that node’s NodeRuntimeOptions::start_timeout if one was set via StartNodeOptions::with_start_timeout.

Default probe timeout is 60 seconds with a 200 ms poll interval; setting SLOW_TEST_ENV=true doubles timeouts. Timeouts fail with a message listing the endpoints that never answered.

App-specific stabilization. After the port probe succeeds during deployment, the local deployer calls wait_readiness_stable(nodes), a hook where an app can wait for cluster-level convergence (membership settled, leader elected) before workloads start. The default is a no-op.

sequenceDiagram
    participant D as Deployer
    participant N as Node process
    D->>N: spawn (config materialized in working dir)
    loop until 2xx or timeout
        D->>N: GET /health/ready
    end
    D->>D: requirement satisfied? (All/Any/AtLeast)
    D->>N: wait_readiness_stable(...)

Static Artifacts and cfgsync

This chapter covers how typed app configs become per-node file artifacts and how containerized backends deliver them to nodes that cannot see your filesystem.

Why cfgsync Exists

The local deployer writes each node’s rendered config into the node’s working directory. Compose and Kubernetes nodes run in containers without access to the host directory where the framework generated those configs. cfgsync transfers the generated per-node files at startup and when a node is restarted with overridden options.

cfgsync consists of a typed artifact model and an HTTP service. A node container starts, registers with the cfgsync server, fetches its artifact set, writes the files locally, and then starts the application. The same artifact types support runtime config overrides through replace_node_artifacts for dynamic node starts on Kubernetes.

sequenceDiagram
    participant R as Runner
    participant S as cfgsync server
    participant C as node container
    R->>S: render config + artifacts, start server
    C->>S: POST /register (identifier, ip, metadata)
    C->>S: POST /node
    S-->>C: NodeArtifactsPayload (files)
    C->>C: write files, exec node binary

The Crates

Protocol (cfgsync-core). NodeRegistration carries a stable identifier, an IPv4 address, and an opaque RegistrationPayload, adapter-owned JSON metadata the framework never interprets (with_metadata(&T) / from_json_str). The server answers /node with a NodeArtifactsPayload (schema version + files) or a structured error: MissingConfig (unknown node), NotReady (registered, artifacts pending), Internal. Client wraps the endpoints: register_node, fetch_node_config, fetch_node_config_status (→ ConfigFetchStatus::{Ready, NotReady, Missing}), and the administrative ReplaceNodeArtifactsRequest for swapping one node’s served files.

Sources. A server serves whatever its NodeConfigSource resolves:

• StaticConfigSource: an in-memory map of identifier → payload, built from payloads or from a NodeArtifactsBundle (per-node entries plus shared_files served to everyone). Registration succeeds only for known identifiers. Supports replace_node_artifacts.
• RegistrationConfigSource<M> (cfgsync-adapter) is registration-aware: it records registrations, snapshots them, and asks a materializer for artifacts on every resolve. Per-node overrides installed via replace_node_artifacts win over materialized files (shared files are still appended).

Materialization (cfgsync-adapter). The adapter contract is one trait:

pub trait RegistrationSnapshotMaterializer: Send + Sync {
    fn materialize_snapshot(
        &self,
        registrations: &RegistrationSnapshot,
    ) -> Result<MaterializationResult, DynCfgsyncError>;
}

RegistrationSnapshot is the current registration set, sorted by identifier for determinism. The result is NotReady (keep polling) or Ready(MaterializedArtifacts): per-node ArtifactSets keyed by identifier plus a shared set appended to every node (resolve(identifier) merges them). Wrappers: CachedSnapshotMaterializer caches results per snapshot, PersistingSnapshotMaterializer additionally pushes ready artifacts into a MaterializedArtifactsSink. A prebuilt MaterializedArtifacts value is itself a materializer that is always ready.

Runtime (cfgsync-runtime). ServerConfig { port, source } loads from YAML; ServerSource is static (serve precomputed artifacts, no registration required) or registration (require registration first). The runtime Client adds local materialization: OutputMap routes artifact paths to disk (OutputMap::under(root), config_and_shared(config_path, shared_dir), or explicit route(...)), and run_client_from_env drives the whole register-fetch-write loop from CFG_SERVER_ADDR, CFG_HOST_IDENTIFIER, CFG_HOST_IP, CFG_REGISTRATION_METADATA_JSON, and output paths (CFG_FILE_PATH, CFG_DEPLOYMENT_PATH). This is what runs inside node containers before the app binary starts.

From Typed Config to Artifacts

The boundary between your typed NodeConfig and cfgsync lives in testing-framework/core/src/cfgsync/mod.rs.

Apps that implement ClusterNodeConfigApplication (see Implementing Application) get StaticNodeConfigProvider for free: build a config for node i, rewrite it for backend hostnames (node-0.svc instead of 127.0.0.1), and serialize it. On top of that:

• build_static_artifacts::<E>(deployment, hostnames) produces a MaterializedArtifacts with one /config.yaml per node-<i> identifier. Hostname count must match the node count.
• render_and_write_registration_server::<E, _>(...) renders both the cfgsync server config YAML and the precomputed artifacts YAML to disk, with an enrich_artifacts hook for app-specific extras (shared files, additional per-node files).
• build_node_artifact_override::<E>(deployment, index, hostnames, options) builds the replacement artifact set for a node started with non-default StartNodeOptions; PeerSelection, config_override, and config_patch are interpreted for container backends here (see node-config.md).

The backends consume these directly: the Compose deployer calls write_registration_server_compose_configs to render the server config and artifacts into the generated stack directory before docker compose up (Compose Deployer); the K8s deployer exposes cfgsync_service, cfgsync_hostnames, and build_cfgsync_override_artifacts hooks on its environment trait and pushes override artifacts through replace_node_artifacts when its manual cluster starts nodes with options (Kubernetes Deployer).

graph LR
    A["ClusterNodeConfigApplication<br/>(typed NodeConfig)"] --> B["build_static_artifacts<br/>(MaterializedArtifacts)"]
    B --> C[cfgsync server]
    C --> D["cfgsync client in container<br/>(writes files)"]
    D --> E[node process]
    C:::pr
    D:::pr
    E:::pr
    classDef pr stroke:#e08a3c,stroke-width:2.5px;

Choosing a Shape

Precomputed (used by the framework deployers): all registrations are known up front, so the deployer materializes every artifact before the stack starts and serves them through a registration-kind source. The rendered cfgsync.artifacts.yaml remains in the stack directory for inspection.

Registration-driven: when artifacts depend on runtime facts (e.g. which IPs registered), implement RegistrationSnapshotMaterializer yourself and return NotReady until the snapshot is complete. The runnable examples in cfgsync/runtime/examples/ (minimal_cfgsync.rs, precomputed_registration_cfgsync.rs, wait_for_registrations_cfgsync.rs) show both shapes end to end.

The framework deployers currently use the precomputed path. The registration-driven materializer is public API for integrations whose per-node configs cannot be finalized before nodes start.

Seeds and Reproducibility

DeploymentSeed controls only part of a run’s variability. This chapter lists what it seeds, what it does not, and what that means for reproducing a run.

DeploymentSeed

DeploymentSeed (testing-framework/core/src/topology/mod.rs) is a 32-byte value:

let seed = DeploymentSeed::new([7u8; 32]);
let bytes: &[u8; 32] = seed.bytes();

The seed exists so generated deployments can be reproduced: record it when a run fails, and the same seed makes the provider return the identical deployment.

You attach it to a scenario with with_deployment_seed:

let scenario = ScenarioBuilder::<KvEnv>::new(provider)
    .with_deployment_seed(DeploymentSeed::new([7u8; 32]))
    .with_run_duration(Duration::from_secs(30))
    .build()?;

The seed has exactly one consumer: when build() resolves the deployment, it calls the deployment provider with it:

pub trait DeploymentProvider<D>: Send + Sync {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

A provider that generates topologies (random shapes, sampled node parameters, derived node ids) should draw all of its randomness from the seed, so the same seed always yields the same deployment. See Topology and Deployment Plans for how providers feed the builder.

What Is Actually Seeded

The current behavior is:

No in-repo deployment provider currently consumes the seed: FixedDeploymentProvider is the only provider shipped, and the example apps all use concrete ClusterTopology values. DeploymentSeed is available to custom generating providers; setting a seed on a fixed deployment has no effect.

Writing a Seeded Provider

A provider that wants reproducible generation reads all of its variability from the seed bytes:

use testing_framework_core::topology::{
    ClusterTopology, DeploymentProvider, DeploymentSeed, DynTopologyError,
};

struct SizedFromSeed {
    min_nodes: usize,
    max_nodes: usize,
}

impl DeploymentProvider<ClusterTopology> for SizedFromSeed {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<ClusterTopology, DynTopologyError> {
        let first = seed.map_or(0, |seed| seed.bytes()[0] as usize);
        let span = self.max_nodes - self.min_nodes + 1;
        Ok(ClusterTopology::new(self.min_nodes + first % span))
    }
}

let scenario = ScenarioBuilder::<KvEnv>::new(Box::new(SizedFromSeed {
        min_nodes: 3,
        max_nodes: 7,
    }))
    .with_deployment_seed(DeploymentSeed::new([7u8; 32]))
    .with_run_duration(Duration::from_secs(30))
    .build()?;

The same seed always produces the same cluster size; omitting the seed uses the provider’s default (seed is None). A generating provider can feed the 32 bytes into a seeded RNG and derive node parameters or NodePlan ids from it.

Practical Reproducibility

• If your provider is seeded, record the seed alongside failures and replay with with_deployment_seed to get the identical deployment.
• Everything downstream of the deployment (ports, PIDs, timing) still varies run to run. Determinism ends at the descriptor; treat expectations accordingly.
• For state-level reproduction (replaying a node from captured state rather than regenerating a topology), use snapshot directories instead; see Persistence, Snapshots, and Recovery Testing.

ManualCluster: Imperative Node Control

ManualCluster provides imperative node lifecycle control. Your code starts, stops, and restarts nodes directly without using the scenario runner.

When to Use It

Use ManualCluster when orchestration lives outside the scenario runtime. Scenarios can also start and restart nodes from workloads through the ClusterHandle; see Chaos and Controlled Failure.

• Step-driven flows: an external driver decides when each node starts and what happens next.
• BDD harnesses: Gherkin steps map naturally onto imperative start/stop/wait calls.
• Exploratory debugging: poke at a live cluster from a main function without writing workloads or expectations.

There are no workloads, expectations, or RunContext; you call methods and assert with your own code.

Creating a Cluster

On the local backend (testing-framework/deployers/local/src/manual/mod.rs), construct one directly from a deployment descriptor:

use testing_framework_runner_local::ManualCluster;

let cluster = ManualCluster::<KvEnv>::from_topology(KvTopology::new(3));

The descriptor defines capacity and indexing, not initial state: no processes exist until you call start_node. E must implement LocalDeployerEnv (see Implementing Application).

On Kubernetes, testing_framework_runner_k8s::ManualCluster is constructed asynchronously because it installs the Helm stack first:

use testing_framework_core::scenario::{ClusterStartMode, DeploymentPolicy, ObservabilityInputs};
use testing_framework_runner_k8s::ManualCluster;

// Install the stack with no nodes running…
let cluster = ManualCluster::<KvEnv>::from_topology(KvTopology::new(3)).await?;

// …or spell out the start mode, policy, and observability inputs
let cluster = ManualCluster::<KvEnv>::provision(
    KvTopology::new(3),
    ClusterStartMode::OnDemand,
    DeploymentPolicy::default(),
    &ObservabilityInputs::default(),
)
.await?;

Manual clusters are the same machinery the managed path uses: a ClusterApp built with .with_start_mode(ClusterStartMode::OnDemand) provisions the identical capacity inside a scenario, leaving node starts to workloads driving the ClusterHandle.

Naming: requested names are normalized to a node- prefix: start_node("a") registers node-a; names already starting with node- pass through; an empty name becomes node-<index>. Each started node needs a fresh name; reusing a registered name is an error.

API

ManualCluster also implements the core NodeControlHandle<E> and ClusterWaitHandle<E> traits, so it can stand behind code written against those abstractions. An app-layer child cluster exposes the same common operations through ClusterHandle<E>, without exposing the backend-specific ManualCluster object.

StartNodeOptions

The full options struct (core/src/scenario/control.rs):

restart_node_with accepts only args and runtime. Passing peers, config_override, config_patch, persist_dir, or snapshot_dir to a restart returns an InvalidArgument error, because a restart reuses the node’s existing config and working directory. To change those, stop the node and start a new one.

Example: Convergence Under Restart

Adapted from the in-repo example cargo run -p kvstore-examples --bin kvstore_k8s_manual_convergence (examples/kvstore/examples/src/bin/k8s_manual_convergence.rs):

let cluster = ManualCluster::<KvEnv>::from_topology(KvTopology::new(3)).await?;

let node0 = cluster.start_node("node-0").await?.client;
let node1 = cluster.start_node("node-1").await?.client;
let node2 = cluster.start_node("node-2").await?.client;
cluster.wait_network_ready().await?;

write_keys(&node0, "kv-manual", 12).await?;
wait_for_convergence(&[node0.clone(), node1.clone(), node2.clone()], "kv-manual", 12).await?;

cluster.restart_node("node-2").await?;
cluster.wait_network_ready().await?;

let node2 = cluster.node_client("node-2").expect("client after restart");
wait_for_convergence(&[node0, node1, node2], "kv-manual", 12).await?;

cluster.stop_all();

The driver determines which nodes exist, when writes happen, what convergence means, and when to inject the restart. write_keys and wait_for_convergence are plain functions over the application’s HTTP client.

That example runs on Kubernetes: the k8s ManualCluster offers the same method surface over pods in a per-run namespace. The local ManualCluster starts processes directly and needs no external infrastructure.

Lifecycle and Cleanup

Dropping the ManualCluster calls stop_all(): every child process is killed and waited on. Node working directories are temporary and removed with the processes unless retained. Set TF_KEEP_LOGS=1 (or true/yes) to keep them for inspection, and see Persistence for deliberate state retention.

External example: logos-blockchain’s cucumber suite drives ManualCluster from Gherkin steps in its own repository, including dependency-ordered starts, targeted restarts, snapshot-on-stop, and restore-from-snapshot.

Persistence, Snapshots, and Recovery Testing

This chapter explains how node working directories behave, what persist_dir and snapshot_dir actually do, and how to build stop/restore recovery tests on top of them.

The Working Directory

Every locally spawned node runs inside a framework-created directory (testing-framework/deployers/local/src/process.rs):

• Launch files (the rendered config) are written into it before spawn, and the process starts with it as its current directory. A node that writes relative paths (a database under ./db, logs under ./logs) keeps all its state there.
• By default the directory is a random-named temp directory created in the current working directory of the test process, and it is deleted when the node is dropped.
• Deletion is skipped when the owning thread is panicking, when the node was spawned with keep_tempdir, when TF_KEEP_LOGS=1 is set, or when the deployment policy sets cleanup_policy.preserve_artifacts (see Readiness, Retry, and Artifact Preservation).

restart (used by ManualCluster::restart_node and LocalProcessHandle::restart) kills the child and respawns it in the same directory with the same launch spec. Launch files are rewritten; everything else is untouched, so state in the working directory survives the restart.

persist_dir: a Predictable Location

persist_dir does not reuse the given path as-is. Verified semantics from create_tempdir:

• The working directory is created as <basename>_<random-suffix> inside the parent of the path you pass. with_persist_dir("/tmp/kv-run/node-0") yields a working directory like /tmp/kv-run/node-0_a1B2c3/. The parent is created if missing.
• Nothing is copied into it; it starts empty apart from launch files.
• It is still a managed temp directory: deleted on drop unless one of the retention switches above applies.

Use persist_dir when a test (or a human) must find the node’s state. Pair it with TF_KEEP_LOGS=1 or preserve_artifacts to keep the directory after the run, then feed it back in as a snapshot later.

snapshot_dir: Seeding State at Start

snapshot_dir copies saved state into the fresh working directory before the process spawns. This is the restore half of a recovery test: a fresh node starts from state captured in an earlier run instead of an empty directory. Verified semantics from copy_snapshot_dir:

• The directory you pass is copied as a subdirectory of the working directory, named after its final path component, with overwrite enabled. with_snapshot_dir("/snapshots/run1/db") produces <workdir>/db/....
• Consequently, the snapshot source’s basename must match the relative path the node expects. If your node reads ./db, snapshot a directory literally named db.
• The copy happens once, at spawn. Restarts do not re-apply it.
• A failed copy fails the spawn (ProcessSpawnError::Snapshot).

The framework copies the supplied directory byte-for-byte without interpreting its contents. The caller determines what constitutes a consistent snapshot, including which directories to copy, whether the node must be stopped first, and whether its on-disk state is crash-consistent.

Where the Options Live

Per dynamic node: StartNodeOptions (full table in ManualCluster):

let node = cluster.start_node_with(
    "restored",
    StartNodeOptions::default()
        .with_snapshot_dir(PathBuf::from("/snapshots/run1/db"))
        .with_start_timeout(Duration::from_secs(90)),
).await?;

Note restart_node_with rejects persist_dir/snapshot_dir overrides, since restarts keep the existing directory. Start a new node to restore from a snapshot.

Per initial node: LocalDeployerEnv::initial_persist_dir(topology, node_name, index) and initial_snapshot_dir(...) (default None). Override these to mount state under the whole initial cluster, e.g. restore every node of a 3-node cluster from a saved dataset before the scenario begins.

Per composed process: LocalProcessApp in the app layer (testing-framework/app/src/process.rs) exposes the same three switches for one-binary apps:

Its LocalProcessHandle offers working_dir(), restart(), stop(), is_running(), pid(), and keep_tempdir(); the process stops when the last handle clone drops (see One Binary: LocalProcessApp).

Recovery-Testing Patterns

Restart in place. State persists because the node reuses its working directory.

write_data(&client).await?;
cluster.restart_node("node-1").await?;
cluster.wait_node_ready("node-1").await?;
assert_data_recovered(&cluster.node_client("node-1").unwrap()).await?;

Stop, snapshot, restore. Full recovery drill via ManualCluster:

// 1. Run a node whose working dir you can locate.
let node = cluster.start_node_with(
    "primary",
    StartNodeOptions::default().with_persist_dir(PathBuf::from("/tmp/kv-run/primary")),
).await?;
write_data(&node.client).await?;

// 2. Stop it, then copy its state out yourself (caller-owned step):
cluster.stop_node("node-primary").await?;
//    e.g. cp -r /tmp/kv-run/primary_*/db /snapshots/case1/db

// 3. Start a fresh node seeded from the snapshot.
let restored = cluster.start_node_with(
    "restored",
    StartNodeOptions::default().with_snapshot_dir(PathBuf::from("/snapshots/case1/db")),
).await?;
cluster.wait_node_ready("node-restored").await?;
assert_data_recovered(&restored.client).await?;

Step 2 is caller-owned because the framework neither snapshots on stop nor knows which files constitute application state. Set TF_KEEP_LOGS=1 so stopped nodes’ directories remain available for copying.

Config continuity. The first node started with a snapshot_dir has its generated config recorded as a template, and that template is passed to the config-build hooks of later dynamic starts (the template_config parameter). An env that honors it can keep restored nodes consistent with the configs their state was produced under.

Cross-run state. Combine persist_dir (findable location) with retention (TF_KEEP_LOGS / preserve_artifacts), archive the directory after run A, and hand it to run B via snapshot_dir or the initial_snapshot_dir hook. Upgrade tests, long-lived-ledger tests, and crash-recovery matrices all reduce to this loop.

Part V — Deployers and Sources

This part covers where scenarios run and where their nodes come from.

Clusters provision to local processes, Docker Compose, or Kubernetes through one request and handle model — the provisioner passed per app selects the backend — and scenarios can also attach to clusters you already operate.

• Capability Matrix — feature support per deployer
• Local Deployer — processes on your machine
• Compose Deployer — containerized clusters
• Kubernetes Deployer — Helm releases in isolated namespaces
• Shared Cluster Provisioning — one request and handle model across cluster sources
• Existing and External Clusters — attaching to live systems
• Binary Providers — resolving node binaries: path, env, build, download
• Readiness, Retry, and Artifact Preservation — deployment policy

Backend Capability Matrix

This page records what each backend provisioner currently supports, based on the provisioner implementations.

The framework ships three cluster provisioners: LocalClusterProvisioner (local processes), ComposeProvisioner (Docker Compose), and K8sClusterProvisioner (Kubernetes/Helm). All three implement the same ClusterProvisioner<E> operation and feed the same scenario runtime; they differ in where nodes run and which runtime surfaces they wire onto the returned ClusterHandle.

Example Application Coverage

This table records concrete adapters and runnable example binaries, rather than what the generic framework could theoretically support.

Row-by-Row

Managed clusters. All three provisioners accept ClusterRequest::managed(deployment) — usually built through ClusterApp — and return a ClusterUnit whose handle carries clients, readiness waits, and cleanup. This is the common path shown in the Local, Compose, and Kubernetes chapters.

On-demand start. Local and k8s prepare the cluster without starting nodes and return a ManualControlled handle whose start_node calls bring nodes up. The compose provisioner rejects on-demand requests with ComposeRunnerError::OnDemandUnsupported.

Node control. The local provisioner attaches the full LocalCluster control surface when the request asks for Full: start, stop, and restart by name, with complete StartNodeOptions support (peer selection, config overrides and patches, persist/snapshot dirs, extra args, timeouts). The compose provisioner wires a ComposeNodeControl handle that supports restart_node via docker compose restart; for attached clusters, ComposeAttachedNodeControl adds stop_node via docker container stop. The k8s provisioner backs control with its ManualCluster, which scales the per-node Deployments between 0 and 1 replicas for start_node, stop_node, restart_node, and wait_node_ready; only default start options are accepted unless the environment implements cfgsync override artifacts. Restarts respawn any kubectl port-forward tunnels on their original local ports so existing clients keep working. Attached k8s clusters get the same scaling-based restart, resolving each node name to its Deployment in the attached namespace at call time; unreachable endpoints are reached through automatically spawned port-forwards.

Attached clusters. ClusterRequest::attached(...) takes descriptors built with ExistingCluster::for_compose_project / for_compose_services (compose) or for_k8s_selector / for_k8s_selector_in_namespace (k8s). The local provisioner rejects attached requests outright. Details in Existing and External Clusters.

External nodes. All three provisioners resolve ExternalNodeSource values into node clients through Application::external_node_client. The local provisioner additionally falls back to a generic endpoint parser (build_external_client) when the application does not override that hook.

Container stacks. ComposeProvisioner also implements ContainerStackProvisioner, so one compose project can hold node clusters and heterogeneous container services declared as ContainerServiceSpec values. Each ContainerServiceHandle supports individual start, stop, restart, readiness, and running-state operations. Kubernetes does not yet implement ContainerStackProvisioner. See Backend Scope.

Observability. Compose and k8s resolve ObservabilityInputs from the LOGOS_BLOCKCHAIN_* env vars merged with the request’s per-cluster values (request wins), pass the OTLP ingest URL into workspace/chart preparation, and store the result on the handle so handle.metrics() returns a Prometheus-backed telemetry handle. The local provisioner leaves observability empty. See Cluster Control and Observability and Telemetry and External Observability.

Manual clusters. testing_framework_runner_local::ManualCluster::from_topology and testing_framework_runner_k8s::ManualCluster::from_topology give imperative code the node-control surface without the scenario runtime. On k8s the same ManualCluster type also backs the managed provisioning path. See ManualCluster.

Binary providers. Binary resolution (PathBinaryProvider, EnvBinaryProvider, BuildBinaryProvider, DownloadBinaryProvider, FallbackBinaryProvider) lives in the local crate and feeds local process launches. Compose and k8s nodes run container images instead, so image selection happens through descriptor specs and env-var overrides, not binary providers. See Binary Providers.

cfgsync artifacts. The compose provisioner writes a cfgsync.yaml into its workspace and can launch a Docker-backed cfgsync config server sidecar; the k8s provisioner supports cfgsync-backed config overrides and cfgsync-rendered bootstrap assets in chart values. The local backend materializes rendered config files directly into each node’s working directory with no cfgsync involvement. See Static Artifacts and cfgsync.

Backend Selection

The local backend runs node processes directly and provides full node control. It requires no infrastructure beyond the node binary, which a binary provider can build. It is the default: with_app(...) uses LocalClusterProvisioner unless with_app_using(...) names another provisioner.

Use Compose for container images, container networking, container-stack composition, or telemetry endpoints. Use Kubernetes to exercise charts, NodePort or port-forward access paths, and cluster infrastructure. Attach to an already-running stack when the cluster outlives the test (see Existing and External Clusters).

Readiness gating, deploy retries, and artifact preservation are controlled uniformly through the DeploymentPolicy carried on each ClusterRequest; see Readiness, Retry, and Artifact Preservation.

Local Backend

LocalClusterProvisioner runs every node as a local OS process. It is the default backend.

The local backend lives in the testing-framework-runner-local crate. It requires no Docker daemon and no cluster: it resolves a node binary, writes each node’s config into a private working directory, spawns the processes, probes readiness, and returns a ClusterHandle backed by the running cluster. with_app(...) uses it implicitly; naming it is equivalent:

use testing_framework_runner_local::LocalClusterProvisioner;

let mut scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<KvEnv>::new(KvTopology::new(3)),
        LocalClusterProvisioner,
    )
    .build()?;

let runner = AppHostDeployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_basic_convergence. No manual binary setup is needed, because kvstore’s fallback provider chain builds the node binary on first use (see Binary Providers).

What provision_cluster Does

For a managed request, LocalClusterProvisioner::provision_cluster:

1. Creates an empty LocalCluster for the deployment (honoring artifact-preservation policy).
2. With the default ClusterStartMode::Eager, spawns one process per topology entry, probes readiness, and retries the whole spawn on failure (see below). With OnDemand, no node starts and the handle’s profile is ManualControlled.
3. Merges any external node sources into the client inventory.
4. Attaches the readiness wait, the cleanup guard, and — when the request asks for ClusterControlRequest::Full — the full node-control surface.

Attached requests are rejected (AttachedUnsupported; attach is a compose/k8s feature, see Existing and External Clusters); external requests produce a clients-only ExternalUncontrolled unit.

The main runtime types are:

• LocalClusterProvisioner: the provisioner. E implements LocalDeployerEnv (full-control hooks) or the compact LocalBinaryApp trait (one binary + one config file + one HTTP port per node).
• LaunchSpec: the launch plan for one process: binary path, files to materialize, CLI args, env vars.
• LocalCluster<E>: the concrete cluster: spawned processes, node manager, clients, and cleanup.

Working Directories and Logs

Each node gets its own temporary working directory, created under the current directory (or under a caller-supplied persist path). Config files and any other LaunchFile entries are written there before spawn, and the process starts with that directory as its cwd.

Node stdout and stderr are inherited from the test process, so node logs interleave with your test output; control verbosity with the RUST_LOG value configured on the app’s LocalProcessSpec (for example .with_rust_log("kvstore_node=info")).

On drop, each node process is killed and its tempdir removed. Two things preserve working directories instead of deleting them:

• DeploymentPolicy with cleanup_policy.preserve_artifacts = true (see Readiness, Retry, and Artifact Preservation), or the TF_KEEP_LOGS=1 env var.
• A panicking test thread, in which case directories are kept automatically for inspection.

Nodes started with a persist_dir or seeded from a snapshot_dir (via StartNodeOptions) copy or place state accordingly before spawn (see Persistence, Snapshots, and Recovery Testing).

Ports

The backend reserves real OS ports up front: allocate_available_port() binds an ephemeral listener and releases it, and reserve_local_node_ports reserves the network port plus any app-named extra ports for each node. Endpoints are surfaced as NodeEndpoints (an API socket address plus named extra ports), from which the app builds its typed NodeClient.

Readiness and Retry

Readiness is governed by the DeploymentPolicy carried on the cluster request (ClusterApp::with_policy):

The probe shape comes from the environment: LocalReadinessProbe::HttpGet { path } (default, using Application::node_readiness_path()) or LocalReadinessProbe::Tcp. If spawn or readiness fails, all nodes from that attempt are dropped and the entire cluster is respawned with exponential backoff and jitter, up to the retry budget.

Node Control

The local backend supports the complete node-control surface. When the request asks for ClusterControlRequest::Full — the ClusterApp default — the handle’s start_node, stop_node, and restart_node operate on nodes by name, with full StartNodeOptions support (peer selection, config overrides and patches, persist and snapshot directories, extra args, start timeouts). The openraft_kv failover scenario uses this path:

cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover

See Cluster Control and Observability for the request fields and handle surface.

Manual Clusters

For orchestration outside the scenario runner, such as Cucumber steps or another test harness, the crate provides an imperative cluster:

use testing_framework_runner_local::ManualCluster;

let cluster = ManualCluster::<KvEnv>::from_topology(descriptors);

cluster.start_node("node-0").await?;
cluster.wait_network_ready().await?;
cluster.stop_all();

ManualCluster exposes start_node(_with), stop_node, restart_node(_with), wait_node_ready, wait_network_ready, node_client, node_pid, node_clients, and add_external_sources / add_external_clients. It is covered in depth in ManualCluster: Imperative Node Control. The same on-demand lifecycle is available inside a scenario via ClusterApp::with_start_mode(ClusterStartMode::OnDemand).

Binary Resolution

Every local node needs an executable. LocalProcessSpec::new("MY_NODE_BIN") defaults to an env-var provider; with_binary_provider swaps in any BinaryProvider, including fallback chains that try an env override first and build with Cargo otherwise. Resolution is cached per process and locked across processes. Full detail in Binary Providers.

The local backend supports external node sources but not attached existing clusters. If Application::external_node_client is not implemented, it falls back to parsing the endpoint (http://host:port) and building a client from the resolved socket address.

Compose Backend

ComposeProvisioner runs each node as a Docker Compose service generated from your deployment descriptor. The same type also provisions heterogeneous container stacks. Each cluster and container stack runs in its own compose project, and every participant attaches its services to one shared session network the provisioner creates lazily — so every service reaches every other by compose service name while owning its file, project, and teardown outright.

The compose backend lives in the testing-framework-runner-compose crate. For a managed cluster it generates a compose file per run, brings the stack up, discovers the host ports Docker assigned, probes readiness, and returns the cluster unit. It requires a running Docker daemon; otherwise provisioning fails with ComposeRunnerError::DockerUnavailable.

use testing_framework_runner_compose::{ComposeProvisioner, ComposeRunnerError};

let mut scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<KvEnv>::new(KvTopology::new(3)),
        ComposeProvisioner::default(),
    )
    .build()?;

let runner = match AppHostDeployer.deploy(&scenario).await {
    Ok(runner) => runner,
    Err(error) if docker_unavailable(&error) => return Ok(()), // skip without Docker
    Err(error) => return Err(error.into()),
};
runner.run(&mut scenario).await?;

(docker_unavailable downcasts the deploy error’s source to ComposeRunnerError::DockerUnavailable; the shipped compose bins show the exact helper.) Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_compose_convergence.

Provisioning Pipeline

flowchart LR
    A[Workspace<br/>tempdir] --> B[Write configs<br/>+ cfgsync.yaml]
    B --> C[Render<br/>compose.generated.yml]
    C --> D[docker compose<br/>create + up]
    D --> E[Port discovery<br/>docker compose port]
    E --> F[Readiness<br/>probes]
    F --> G[Node clients<br/>+ ClusterUnit]

1. Workspace. A temporary ComposeWorkspace is created; the app’s ComposeDeployEnv::prepare_compose_configs writes per-node config files (for ComposeBinaryApp environments, one static config per node under stack/configs/, rewritten for service hostnames node-0, node-1, …).
2. cfgsync. If the environment enables ComposeConfigServerMode::Docker, a cfgsync config server container is started on an ephemeral port and the provisioner waits for it to accept TCP connections before proceeding. The default mode is Disabled. See Static Artifacts and cfgsync.
3. Compose file. The env’s compose_descriptor (image, entrypoint, volumes, ports, environment, optional platform per service) is rendered through the Tera template at testing-framework/deployers/compose/assets/docker-compose.yml.tera into compose.generated.yml. The template is resolved relative to the repository root (CARGO_WORKSPACE_DIR override respected). Required images are checked with docker image inspect up front. The provisioner never builds or pulls them; a missing image fails the deploy with MissingImage.
4. Bring-up. docker compose create and docker compose up run under a unique project name (compose-stack-<uuid>). On failure, container logs are dumped before cleanup.
5. Ports. Container ports map to ephemeral host ports; the provisioner resolves each with docker compose port and records them as NodeHostPorts { api, testing }. The host defaults to 127.0.0.1 and can be overridden with COMPOSE_RUNNER_HOST.
6. Readiness. Per the env’s ComposeReadinessProbe: HTTP GET against Application::node_readiness_path() on each mapped API port, or raw TCP reachability. Gated by DeploymentPolicy.readiness_enabled; when disabled, the stack gets a short fixed grace period instead. See Readiness, Retry, and Artifact Preservation.
7. Clients. build_node_client runs against the discovered host/port pairs, producing the cluster’s typed node clients; external node sources are appended through external_node_client.

Retry. When the request’s DeploymentPolicy carries a retry_policy, the whole managed provisioning attempt is retried with exponential backoff and jitter up to the configured attempts; without one, provisioning is a single attempt.

Start mode. ClusterStartMode::OnDemand is not supported: the provisioner rejects it with ComposeRunnerError::OnDemandUnsupported.

Node Control

When the request asks for ClusterControlRequest::Full — the ClusterApp default — the provisioner attaches a ComposeNodeControl handle bound to the generated compose file and project. It supports restart only: restart_node(name) shells out to docker compose restart <service>. Start and stop of individual node services are not wired for managed compose clusters. The openraft_kv failover scenario runs on this backend: cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover.

Container Stacks

ComposeProvisioner also implements ContainerStackProvisioner, the heterogeneous-service contract from testing-framework-container. An app declares ContainerServiceSpec values and deploys them through DeployContext::deploy_container_stack. Each stack runs as its own compose project attached to the provisioner’s shared session network, so an app can prepare container dependencies in order and use their internal endpoints when declaring dependents.

Node clusters attach to the same network, in either deploy order. A stack that mixes clusters and container services therefore has compose-internal DNS between them — a sidecar can reach node-0 by service name without going through published host ports — while each participant keeps its own project: tearing one down never touches the others’ files or containers. The network is labeled testing-framework=session; the last participant to leave removes it, and a crashed process can leak it (prune with docker network rm on that label).

A session holds any number of managed clusters as long as they are named. ClusterRequest::with_name("alpha") (or ClusterApp::with_name) namespaces that cluster’s services, config hostnames, and config files as alpha-node-0, alpha-node-1, …, and gives it its own cfgsync configuration; an unnamed cluster keeps the plain node-N names. One cluster per session may stay unnamed — a second unnamed request fails with an error pointing at with_name, a duplicate name fails with ClusterAlreadyProvisioned, and a service name colliding with any registered participant fails with ServiceNameConflict. Cluster names must be short DNS labels. Terminal errors (duplicate clusters, name conflicts, invalid names, unsupported on-demand start) are never retried, so the first real failure propagates instead of burning the retry budget.

Preservation is aggregated at session level and independent of deploy order: when any participant requests preserve_artifacts (or the COMPOSE_RUNNER_PRESERVE/TESTNET_RUNNER_PRESERVE env vars are set), every participant’s cleanup skips teardown, persists its workspace, leaves cfgsync running, and keeps the session network.

Published runner ports are allocated explicitly and remain stable across container lifecycle changes. Each ContainerServiceHandle supports start, stop, restart, wait_ready, and is_running. The complete examples are cargo test -p multi-app-e2e --test compose_happy_path and, for mixed cluster-plus-container sessions in both orders, cargo test -p multi-app-e2e --test compose_mixed_cluster.

Stack containers default to ContainerRestartPolicy::Never, keeping crashes visible to workloads and expectations consistently across backends. An app can request OnFailure or Always explicitly when automatic recovery is part of the behavior under test. Managed node-cluster services use an on-failure policy.

Attaching to an Existing Stack

The compose provisioner fully supports attached clusters. A request built with ClusterRequest::attached(ExistingCluster::for_compose_project("my-project")) skips workspace generation entirely: services are discovered from the running project (or taken from for_compose_services), each container’s labeled API port is inspected, and clients are built through Application::external_node_client. Attached node control gains stop_node in addition to restart_node, implemented with docker container stop / docker container restart against discovered container IDs. A service name listed in for_compose_services that does not exist in the project fails attachment with an error naming the unknown and available services, and an attached readiness wait whose requested services are all stopped fails instead of returning instantly. A managed compose cluster records its own project and node service names as an ExistingCluster attachment exposed through ClusterHandle::attachment(), so deploy-then-attach flows do not need hand-written identifiers. See Existing and External Clusters.

Observability

The provisioner resolves ObservabilityInputs by merging the LOGOS_BLOCKCHAIN_METRICS_QUERY_URL, LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL, and LOGOS_BLOCKCHAIN_GRAFANA_URL env vars with the request’s per-cluster values (request values win). The OTLP ingest URL is passed into config preparation so node configs can point at your collector; the resolved inputs land on the cluster handle, whose metrics() returns a Prometheus-backed handle. Setting TESTNET_PRINT_ENDPOINTS prints Prometheus/Grafana endpoints and per-node pprof profile URLs to stdout. See Telemetry and External Observability.

Cleanup

The cluster’s cleanup guard runs docker compose down, shuts down the cfgsync container if one was started, and removes the workspace. Setting COMPOSE_RUNNER_PRESERVE (or TESTNET_RUNNER_PRESERVE) keeps the stack running and persists the workspace directory for post-mortem inspection; the preserved path is logged.

Requirements recap:

Kubernetes Backend

K8sClusterProvisioner installs each cluster as a Helm release in a throwaway namespace and reaches nodes through NodePorts or port-forwards.

The Kubernetes backend lives in the testing-framework-runner-k8s crate. It talks to whatever cluster your current kubeconfig context points at (kube::Client::try_default()), installs a Helm release, waits for the workloads, and builds node clients against externally reachable ports. Managed provisioning is built on the same ManualCluster unit that imperative tests use — one lifecycle implementation for both.

use testing_framework_runner_k8s::{K8sClusterProvisioner, ManualClusterError};

let mut scenario = AppHost::scenario()
    .with_app_using(
        ClusterApp::<KvEnv>::new(KvTopology::new(3)),
        K8sClusterProvisioner,
    )
    .build()?;

let runner = match AppHostDeployer.deploy(&scenario).await {
    Ok(runner) => runner,
    Err(error) if k8s_unavailable(&error) => return Ok(()), // no cluster available
    Err(error) => return Err(error.into()),
};
runner.run(&mut scenario).await?;

(k8s_unavailable downcasts the deploy error’s source to ManualClusterError::ClientInit, or an InstallStack whose message indicates an unreachable API server; the shipped k8s bins show the exact helper.) Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_k8s_convergence.

Charts and Values

The environment trait K8sDeployEnv produces installable assets via prepare_assets, returning a PreparedK8sStack. Two asset shapes exist:

• Generated single-template charts. Apps implementing K8sBinaryApp get the standard shape for free: render_binary_config_node_manifest renders one ConfigMap (the serialized node config), one Deployment (single replica, --config arg, config mounted from the ConfigMap), and one NodePort Service per node, then render_manifest_chart_assets wraps them in a minimal chart (RenderedHelmChartAssets).
• Real chart directories. NodeRuntimeSpec builds RunnerChartValues (node image, pull policy, fullname override, asset mount layout, node group, optional shared bootstrap service with cfgsync configs and start scripts) and a HelmReleaseBundle with --set values and --set-file entries for start scripts and bootstrap configs. RunnerAssetLayout fixes where bootstrap configs and runner scripts land inside the chart’s mount path.

Node images are resolved from env vars: for a conventional BinaryConfigK8sSpec the primary override is <APP>_K8S_IMAGE, the fallback <APP>_IMAGE, and the default <binary-name>:local with imagePullPolicy: IfNotPresent.

Each run installs into fresh identifiers: namespace tf-testnet-<timestamp>-<pid>, release tf-runner (override via K8sDeployEnv::cluster_identifiers).

Lifecycle Waits

After helm install, an eagerly started cluster waits in stages:

1. Deployment readiness: each node Deployment must report ready replicas (timeout K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS, default 180 s).
2. Port discovery: each node Service must have allocated NodePorts for the API and auxiliary ports declared by collect_port_specs.
3. HTTP readiness: nodes are probed over their NodePorts at node_readiness_path(). The probe host is K8S_RUNNER_NODE_HOST if set, else KUBERNETES_SERVICE_HOST, else 127.0.0.1. If NodePort probing fails (common when the cluster’s node IPs are not routable from the runner), the backend transparently falls back to kubectl port-forward per service and probes over 127.0.0.1.
4. Policy-gated cluster readiness: a final probe pass controlled by the request’s DeploymentPolicy.readiness_enabled / readiness_requirement. See Readiness, Retry, and Artifact Preservation.

HTTP wait tuning: K8S_RUNNER_HTTP_TIMEOUT_SECS (default 240), K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS (default 30), K8S_RUNNER_HTTP_POLL_INTERVAL_SECS (default 1).

Node Control and Start Modes

When the request asks for ClusterControlRequest::Full — the ClusterApp default — the handle supports start_node, stop_node, restart_node, and wait_node_ready, implemented by patching each per-node Deployment’s replicas between 0 and 1 and waiting for the rollout. It works over both direct NodePorts and the kubectl port-forward fallback — after a restart or start the node’s forwards are respawned on their original local ports, so existing clients keep working. Two k8s-specific limits on start_node_with(StartNodeOptions): persist_dir / snapshot_dir are rejected, and peer selection or config overrides require the environment to implement cfgsync override artifacts (cfgsync_service + build_cfgsync_override_artifacts); the override is pushed to the in-cluster cfgsync service through a temporary port-forward before the node starts.

ClusterStartMode maps directly onto the lifecycle: Eager installs and starts every node before the handle is returned (FrameworkManaged); OnDemand installs the release with all node Deployments scaled to zero so your code decides when each node starts (ManualControlled). cargo run -p kvstore-examples --bin kvstore_k8s_restart_chaos exercises managed control end to end with the restart chaos workload (Chaos and Controlled Failure).

Attached clusters get the same restart control: node names resolve to their Deployments in the attached namespace at call time, so a cluster whose workloads do not follow the one-Deployment-per-node convention fails per call with an error naming the deployment and namespace. Only default restart options are accepted (attached clusters have no cfgsync), and port-forwarded access respawns the affected forward after each restart.

Imperative Use: ManualCluster

The same unit is available directly for code that owns the sequence:

use testing_framework_runner_k8s::ManualCluster;

let cluster = ManualCluster::<OpenRaftKvEnv>::from_topology(OpenRaftKvTopology::new(3)).await?;

cluster.start_node("node-0").await?;
cluster.start_node("node-1").await?;
cluster.wait_network_ready().await?;
cluster.restart_node("node-0").await?;
cluster.stop_all();

from_topology is the on-demand default; ManualCluster::provision(topology, start_mode, policy, &observability) selects eager start, a non-default DeploymentPolicy, or explicit observability inputs. The failover demonstration uses this path end to end: cargo run -p openraft-kv-examples --bin openraft_kv_k8s_failover. Contrast with the local variant in ManualCluster: Imperative Node Control.

Attaching to an Existing Cluster

Attached requests are supported with a k8s descriptor: ClusterRequest::attached(ExistingCluster::for_k8s_selector("app.kubernetes.io/instance=tf-runner")) (optionally namespaced with for_k8s_selector_in_namespace). Services matching the selector are listed, each service’s single TCP NodePort (or the port named http/api) becomes the node endpoint, and clients are built via Application::external_node_client. When endpoints are not reachable from the runner, discovery falls back to kubectl port-forward per service. The handle carries a readiness wait, and requesting Full control wires deployment-scaling restart just like managed clusters. See Existing and External Clusters.

Observability and Cleanup

Observability inputs resolve exactly as in compose (LOGOS_BLOCKCHAIN_* env vars merged with the request’s per-cluster values), and TESTNET_PRINT_ENDPOINTS prints Prometheus/Grafana and per-node pprof endpoints. Cleanup uninstalls the Helm release and deletes the namespace (Kubernetes API first, kubectl delete namespace fallback), after killing any port-forward processes. Set K8S_RUNNER_PRESERVE to keep the release and namespace for inspection.

There is no k8s ContainerStackProvisioner yet: heterogeneous container stacks run on the compose backend (Backend Scope).

Requirements recap:

Cluster Provisioning

Every node cluster in the framework is provisioned through one model. A ClusterRequest<E> describes the cluster source and required behavior; a backend ClusterProvisioner<E> turns it into a ClusterUnit<E>; applications and workloads use the resulting backend-neutral ClusterHandle<E>. ClusterApp is a thin builder over this model, and ManualCluster reuses the same lifecycle machinery imperatively.

One Request, Three Sources

ClusterRequest<E> separates what the test needs from how a backend supplies it:

let managed = ClusterRequest::<QueueEnv>::managed(QueueTopology::new(3));
let attached = ClusterRequest::<QueueEnv>::attached(existing_cluster);
let external = ClusterRequest::<QueueEnv>::external(node_sources);

Managed and attached sources can also include external nodes with with_external_nodes(...). This is useful when one logical cluster combines framework-visible nodes from more than one source. See Existing and External Clusters for the attach and external arms in detail.

Requesting Behavior

The request carries requirements that are meaningful across backends:

Backends support different combinations of these requirements — for example, compose rejects OnDemand. The Backend Capability Matrix records current coverage.

Provisioning Inside an AppDeployment

DeployContext is parameterized by a provisioner type. Its deploy_cluster method is the app-layer entry point:

#[async_trait]
impl AppDeployment<AppHostEnv> for QueueLocalApp {
    type Handle = ClusterHandle<QueueEnv>;

    async fn deploy(
        self,
        ctx: &mut DeployContext<AppHostEnv>,
    ) -> Result<Self::Handle, DynError> {
        ctx.deploy_cluster(ClusterRequest::<QueueEnv>::managed(self.deployment))
            .await
    }
}

The request is forwarded unchanged; runtime control is granted only when the request asks for it. deploy_cluster also moves the unit’s cleanup guard into the scenario cleanup stack before returning the handle.

For the common cases, ClusterApp<E> builds the request for you — ClusterApp::new(topology) is a managed, eager, full-control request, with with_policy, with_start_mode, with_control, with_external_nodes, and with_observability as overrides — and is itself an AppDeployment whose handle is ClusterHandle<E>, so single-cluster scenarios need no custom deployment at all.

with_app(app) selects the default LocalClusterProvisioner. with_app_using(app, provisioner) supplies another provisioner; the root deployment must implement AppDeployment<E, P> for that provisioner type (code written only as AppDeployment<E> uses the default local type, while ClusterApp is generic over any ClusterProvisioner<E>).

The Returned Handle

ClusterHandle<E> presents the common runtime surface:

• clients: node_clients, clients, first_client, node_client(name), node_pid(name);
• cluster description: deployment, node_count, control_profile;
• node operations when present: start_node(_with), stop_node, restart_node(_with), wait_node_ready;
• cluster readiness: wait_network_ready;
• telemetry: observability() and metrics(), which builds the Prometheus-backed Metrics handle from the cluster’s observability inputs.

Unavailable operations return an error (“cluster node control is not available”, “cluster readiness is not available”) or None; callers can inspect control_profile() when behavior depends on ownership mode.

The handle does not own managed lifetime. The provisioner returns a cleanup guard inside the unit; DeployContext moves that guard into the scenario cleanup stack, which runs in reverse acquisition order on normal completion and partial deployment failure.

Backend Boundary

ClusterProvisioner<E> has one operation:

#[async_trait]
pub trait ClusterProvisioner<E: Application>: Clone + Send + Sync + 'static {
    async fn provision_cluster(
        &self,
        request: ClusterRequest<E>,
    ) -> Result<ClusterUnit<E>, DynError>;
}

A backend implementation translates the request into concrete resources, clients, control adapters, readiness, and cleanup. ClusterUnit<E> carries those values from the provisioner — the node clients, a ClusterControlProfile, and optional node control, cluster wait, cleanup guard, and observability inputs — and unit.handle() derives the ClusterHandle applications actually use.

Three implementations ship:

Heterogeneous container services use the separate ContainerStackProvisioner contract (DeployContext::deploy_container_stack) because their units are mixed containers, not one typed uniform cluster.

Relation to Entry Patterns

• A single-cluster scenario wraps one request in a ClusterApp.
• A composed stack asks its DeployContext to provision one or more child clusters, each with its own request.
• An attached or external test changes the request’s ClusterSource, while workloads keep using clients and available controls.
• ManualCluster uses the same provisioning machinery — on k8s it is the managed unit — and gives imperative code responsibility for sequencing; inside the model, ClusterStartMode::OnDemand provides the same lifecycle behind a ClusterHandle.

The entry patterns differ in who describes and drives the test. They do not need separate definitions of what a cluster is, which controls it exposes, or who tears it down.

See Also

• Existing and External Clusters: the attach and external request arms.
• AppDeployment and DeployContext: composing cluster and process children.
• Handle Ownership and Teardown: the lifetime boundary in detail.
• Readiness, Retry, and Artifact Preservation: the policies carried by a request.

Existing and External Clusters

Scenarios can run against nodes the framework did not deploy: an attached existing cluster, standalone external endpoints, or a mix.

Where a cluster’s nodes come from is one field of its ClusterRequest. The three arms of ClusterSource cover every case: managed nodes the provisioner spawns, attached nodes discovered in an existing cluster, and external nodes named by static endpoints. Workloads and expectations see the same ClusterHandle either way.

The Source Model

The source model uses these types from testing-framework-core:

ExistingCluster is constructed with for_k8s_selector(selector), for_k8s_selector_in_namespace(namespace, selector), for_compose_project(project), or for_compose_services(project, services). ExternalNodeSource::new(label, endpoint) wraps a plain endpoint string.

Each source arm maps to a ClusterControlProfile on the returned handle, which workloads can consult to know whether the framework owns node lifecycles (framework_owns_lifecycle() is true only for FrameworkManaged).

Building the Requests

use testing_framework_core::scenario::{ClusterRequest, ExistingCluster, ExternalNodeSource};

// Attach to a running compose project instead of deploying nodes.
let attached = ClusterRequest::<KvEnv>::attached(
    ExistingCluster::for_compose_project("compose-stack-1234".into()),
);

// A cluster made only of static endpoints.
let external = ClusterRequest::<KvEnv>::external(vec![ExternalNodeSource::new(
    "staging-gateway".into(),
    "http://staging.example.net:8080".into(),
)]);

// A managed cluster with one extra external endpoint in its inventory.
let hybrid = ClusterApp::<KvEnv>::new(KvTopology::new(2)).with_external_nodes(vec![
    ExternalNodeSource::new("fixed-peer".into(), "http://10.0.0.5:8080".into()),
]);

External nodes compose with every arm: managed + external and attached + external are both valid hybrids, useful when one logical cluster combines framework-visible nodes from more than one source.

Inside an AppDeployment, any of these requests goes through ctx.deploy_cluster(request); the simplest scenarios wrap them via ClusterApp and with_app_using with the matching backend provisioner.

From Source to Typed Client

External and attached sources become typed clients through one hook on the Application trait:

fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;

The default implementation errors with “external node sources are not supported”; an application opts in by parsing source.endpoint() and constructing its client. The local provisioner additionally falls back to a generic parser that resolves http://host:port endpoints and builds the client from the socket address when the app has not overridden the hook.

Per-Backend Support

• Local (LocalClusterProvisioner): no attach — attached requests are rejected with AttachedUnsupported. External nodes are supported.
• Compose (ComposeProvisioner): attach requires a compose descriptor. Services are taken from the descriptor or discovered from the running project; each container’s labeled API port is inspected and turned into an ExternalNodeSource fed to external_node_client. The handle gets a readiness wait, and — when the request asks for ClusterControlRequest::Full — node control supporting restart_node and stop_node against the discovered container IDs. An attach that resolves zero clients fails the deploy. See Compose Backend.
• K8s (K8sClusterProvisioner): attach requires a k8s descriptor. Services matching the label selector are listed in the namespace (default default); each service’s single TCP NodePort (preferring ports named http or api) becomes the endpoint. The handle gets a readiness wait but no node control — that is a current gap; drive lifecycle on k8s through managed clusters or ManualCluster instead. See Kubernetes Backend.
• External requests are supported by all three provisioners and always produce an ExternalUncontrolled handle: clients only, no control, no readiness wait, no teardown.

Use Cases

• Staging and live networks. Point ClusterRequest::external at long-lived endpoints and run workloads and expectations against them; the framework never touches their lifecycle (ExternalUncontrolled).
• Shared test stacks. Deploy a compose or k8s stack once, keep it alive with the backend preserve env vars (see Readiness, Retry, and Artifact Preservation), and attach many fast scenarios to it by project name or label selector.
• Hybrid clusters. Combine managed nodes with an external dependency, for example a locally deployed cluster that must interoperate with a fixed remote peer, via with_external_nodes.

Manual clusters have their own external hooks: add_external_sources and add_external_clients on ManualCluster merge external endpoints into an imperatively driven cluster (see ManualCluster).

Binary Providers

Binary providers resolve the executable a local node process runs. The source can be a path, an env var, a build command, a download, or an ordered fallback chain.

Every local node launch has exactly one provider selected on its LocalProcessSpec. Providers live in testing_framework_runner_local::binary and are re-exported from the crate root. They apply to the Local Deployer only; compose and k8s nodes run container images instead (see the Capability Matrix).

The Trait

pub trait BinaryProvider: Send + Sync {
    fn try_resolve(&self) -> Result<Option<PathBuf>, BinaryProviderError>;
    fn display(&self) -> String;
    fn cache_key(&self) -> String;

    // Provided: cache lookup, then resolve_uncached.
    fn resolve(&self) -> Result<PathBuf, BinaryProviderError> { /* ... */ }
    fn resolve_uncached(&self) -> Result<PathBuf, BinaryProviderError> { /* ... */ }
}

pub type BinaryProviderRef = Arc<dyn BinaryProvider>;

try_resolve returns Ok(None) when the provider is valid but cannot produce a binary in the current environment, which is not an error. Standalone, resolve turns None into BinaryProviderError::NotFound; inside a FallbackBinaryProvider, None means “try the next provider”. Other errors (a failed build, a checksum mismatch) abort resolution immediately.

The Providers

PathBinaryProvider::new(path): a deterministic explicit path. No filesystem search, no PATH lookup.

EnvBinaryProvider::new("MY_NODE_BIN"): the standard override hook. LocalProcessSpec::new(env_var) installs one of these by default.

BuildBinaryProvider delegates to any command:

BuildBinaryProvider {
    command: BuildCommand::new("cargo").with_args(["build", "-p", "kvstore-node"]),
    output_path: "target/debug/kvstore-node".into(), // relative to working_dir
    working_dir: Some(workspace_root),                // default: current dir
    lock_dir: None,                                   // default: <working_dir>/.tf-binaries
}

The command is not Cargo-specific; it can invoke a Make target, shell script, or cache fetch. After the command succeeds, the configured output_path must exist or resolution fails with MissingBuildOutput.

DownloadBinaryProvider fetches into a cache directory (default target/.tf-binaries under the current directory):

• DownloadUrl::Fixed(url) or DownloadUrl::Env(var) selects the source.
• DownloadChecksum::Fixed(sha256) or DownloadChecksum::Env(var) enables SHA-256 verification; mismatches fail with ChecksumMismatch before anything is written to the final path.
• A DownloadProcessor post-processes artifacts that are not directly executable (archives, bundles). It receives the verified download and must materialize the executable at the output path. DownloadProcessorFn::new(cache_key, closure) (or .with_processor_fn(...)) is the lightweight adapter; the cache_key is part of cache identity, so changing your extraction logic invalidates previously prepared binaries.
• On Unix, the result is marked executable (0o755). Downloads are staged through temporary .download/.part files and renamed into place.

FallbackBinaryProvider::new([a, b, ...]): an ordered chain, tried first to last. From the launch spec’s perspective it is still a single provider.

Caching and Cache Identity

Successful resolutions are cached per process in a global map keyed by cache_key(), so repeated node starts with the same provider config do not rebuild, redownload, or re-scan. Cache identity encodes the full request:

• path:<path>, env:<var>
• build:<command>:<output_path>:<working_dir>
• download:<url-or-env>:<checksum-or-env>:<processor-key>:<cache_dir>
• fallback: the members’ keys joined with commas

Change any component and you get a fresh resolution. The download provider also caches on disk: the cached file name hashes the URL, resolved checksum, and processor key, so an already-downloaded binary is reused across processes without refetching.

Concurrent Resolution Locking

Builds and downloads may be triggered by several test processes at once (e.g. cargo nextest running integration tests in parallel). Providers that materialize files take a cross-process file lock before doing work: a lock file created with create_new under .tf-binaries (build) or the download cache dir, retried every 200 ms for up to 10 minutes, then BinaryProviderError::LockTimeout. The lock file is removed when the guard drops.

A killed test process can leave a stale lock file behind. If resolution hangs and then times out, look for leftover *.lock files under .tf-binaries and delete them.

Worked Example: kvstore’s Fallback Chain

From examples/kvstore/testing/integration/src/local_env.rs, which prefers an explicit env override and otherwise builds from source:

use std::{path::PathBuf, sync::Arc};
use testing_framework_runner_local::{
    BinaryProviderRef, BuildBinaryProvider, BuildCommand, EnvBinaryProvider,
    FallbackBinaryProvider, LocalProcessSpec,
};

fn kvstore_binary_provider() -> FallbackBinaryProvider {
    let providers: [BinaryProviderRef; 2] = [
        Arc::new(EnvBinaryProvider::new("KVSTORE_NODE_BIN")),
        Arc::new(BuildBinaryProvider {
            command: BuildCommand::new("cargo")
                .with_args(["build", "-p", "kvstore-node", "--bin", "kvstore-node"]),
            output_path: PathBuf::from(format!(
                "target/debug/kvstore-node{}",
                std::env::consts::EXE_SUFFIX
            )),
            working_dir: Some(workspace_root()),
            lock_dir: None,
        }),
    ];
    FallbackBinaryProvider::new(providers)
}

fn local_process_spec() -> LocalProcessSpec {
    LocalProcessSpec::new("KVSTORE_NODE_BIN")
        .with_binary_provider(kvstore_binary_provider())
        .with_rust_log("kvstore_node=info")
}

First run: KVSTORE_NODE_BIN is unset, the env provider yields None, the build provider compiles the node under the workspace lock, and the path is cached for the rest of the process. Set KVSTORE_NODE_BIN=/path/to/kvstore-node to skip the build entirely, which is useful for prebuilt release binaries or mixed-version clusters (via local_process_spec_for_node, see Local Deployer).

Readiness, Retry, and Artifact Preservation

DeploymentPolicy is the single policy struct that controls readiness gating, deploy retries, and artifact retention across all backends. It travels per cluster on the ClusterRequest.

The Policy

From testing-framework-core (core/src/scenario/deployment_policy.rs):

pub struct DeploymentPolicy {
    pub readiness_enabled: bool,
    pub readiness_requirement: HttpReadinessRequirement,
    pub retry_policy: Option<RetryPolicy>,
    pub cleanup_policy: CleanupPolicy,
}

pub struct RetryPolicy {
    pub max_attempts: usize,
    pub base_delay: Duration,
    pub max_delay: Duration,
}

pub struct CleanupPolicy {
    pub preserve_artifacts: bool,
}

Defaults: readiness_enabled: true, readiness_requirement: HttpReadinessRequirement::AllNodesReady, retry_policy: None, preserve_artifacts: false. HttpReadinessRequirement is AllNodesReady, AnyNodeReady, or AtLeast(usize).

Set it per cluster with ClusterApp::with_policy (or ClusterRequest::with_policy inside a custom AppDeployment):

use std::time::Duration;
use testing_framework_app::ClusterApp;
use testing_framework_core::scenario::{
    CleanupPolicy, DeploymentPolicy, HttpReadinessRequirement, RetryPolicy,
};

let scenario = AppHost::scenario()
    .with_app(ClusterApp::<KvEnv>::new(KvTopology::new(3)).with_policy(
        DeploymentPolicy {
            readiness_enabled: true,
            readiness_requirement: HttpReadinessRequirement::AtLeast(2),
            retry_policy: Some(RetryPolicy::new(
                5,
                Duration::from_millis(500),
                Duration::from_secs(5),
            )),
            cleanup_policy: CleanupPolicy::new(true),
        },
    ))
    .build()?;

Because the policy is carried per request, two clusters in one composed stack can run under different readiness and retry regimes.

Readiness

readiness_enabled and readiness_requirement gate the post-provision probe pass in every backend provisioner. The probe shape (HTTP path vs TCP) comes from the application environment; see the per-backend chapters (Local, Compose, K8s).

Retry

retry_policy drives the local provisioner’s spawn-and-readiness loop: on failure, all nodes from the attempt are dropped and the cluster is respawned with exponential backoff (from base_delay, capped at max_delay, with jitter) up to max_attempts. When retry_policy is None, the local provisioner falls back to its built-in default of 3 attempts, 250 ms base delay, 2 s max delay.

The Compose provisioner honors retry_policy when it is set, re-provisioning the managed project with the same backoff shape; when it is None, a Compose failure is returned on the first attempt. The Kubernetes provisioner honors the readiness fields but does not repeat deployment on failure.

Artifact Preservation

cleanup_policy.preserve_artifacts controls artifact and tempdir retention, not teardown ordering. Teardown itself always follows the runner’s cleanup-guard chain (see Handle Ownership and Teardown); this flag only decides whether per-node working directories survive it.

The local orchestrator computes retention as:

policy.cleanup_policy.preserve_artifacts || keep_tempdir_from_env() // TF_KEEP_LOGS

so either the policy flag or TF_KEEP_LOGS=1 (also true/yes) keeps every node’s working directory (configs, on-disk state, anything the process wrote) after the run. Panicking tests preserve working directories regardless.

Compose honors the policy at session level: any participant requesting preserve_artifacts (or the COMPOSE_RUNNER_PRESERVE / TESTNET_RUNNER_PRESERVE env vars) keeps the whole shared session — stack, workspace, and cfgsync — regardless of which participant established it. Kubernetes preserves through its env var only: K8S_RUNNER_PRESERVE keeps the Helm release and namespace. See Diagnostics and Retained Artifacts.

Part VI — Extending and Reference

This part documents the extension points, the crate map, and the boundary rules that keep the framework application-agnostic.

• Public Extension Points — the traits you implement to plug in
• Crate and API Map — which concept lives in which crate
• Framework vs Application Boundaries — what belongs where, and how it is enforced

Public Extension Points

This chapter lists every trait you implement to plug your application into the framework.

The framework never imports your application. It defines public traits that your integration crate implements and calls them at defined points in the run lifecycle. Each entry below links to the corresponding chapter.

Environment and Topology

Application is the root of every integration. It bundles the backend-specific types the scenario engine is generic over: a deployment descriptor, a node client, and a node config. The three methods have working defaults: override external_node_client to support external sources, build_node_client to support deployer-discovered nodes, and node_readiness_path when your health endpoint is not /.

#[async_trait]
pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str; // default "/"
}

Plugs in as the E type parameter of ScenarioBuilder<E>, Workload<E>, Expectation<E>, and every cluster provisioner.

DeploymentProvider<D> builds the deployment descriptor a scenario runs against, optionally driven by a DeploymentSeed for reproducible generation. ScenarioBuilder::new accepts one; ScenarioBuilder::with_deployment wraps a fixed value in the built-in FixedDeploymentProvider.

pub trait DeploymentProvider<D: DeploymentDescriptor>: Send + Sync {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

Scenario Behavior

Workload<E> describes an action sequence executed during the run. start receives the RunContext<E> (node clients, extensions, run metrics) and runs concurrently with other workloads. A workload can bundle its own checks via expectations().

#[async_trait]
pub trait Workload<E: Application>: Send + Sync {
    fn name(&self) -> &str;
    fn expectations(&self) -> Vec<Box<dyn Expectation<E>>> { Vec::new() }
    fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) }
    async fn start(&self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

Registered with with_workload / with_workload_boxed on the builder.

Expectation<E> defines a check evaluated during or after the run. start_capture records a baseline, check_during_capture is the optional fail-fast hook polled during the run, and evaluate delivers the verdict at the end.

#[async_trait]
pub trait Expectation<E: Application>: Send + Sync {
    fn name(&self) -> &str;
    fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) }
    async fn start_capture(&mut self, ctx: &RunContext<E>) -> Result<(), DynError> { Ok(()) }
    async fn check_during_capture(&mut self, ctx: &RunContext<E>) -> Result<(), DynError> { Ok(()) }
    async fn evaluate(&mut self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

Registered with with_expectation / with_expectation_boxed.

RuntimeExtensionFactory<E> prepares one typed value after deployment (node clients are available) and before workloads start. The value is stored by TypeId and retrieved in workloads via ctx.extension::<T>() / ctx.require_extension::<T>(). Return PreparedRuntimeExtension::new(value), ::with_cleanup(value, guard), or ::from_task(value, join_handle) to tie a background task’s lifetime to the run.

#[async_trait]
pub trait RuntimeExtensionFactory<E: Application>: Send + Sync {
    async fn prepare(
        &self,
        deployment: &E::Deployment,
        node_clients: NodeClients<E>,
    ) -> Result<PreparedRuntimeExtension, DynError>;
}

Registered with with_runtime_extension_factory. Registering two factories that produce the same extension type fails at prepare time with duplicate runtime extension type registered.

Observation

Observer owns the app-side logic of the continuous observation runtime: init builds retained state from the source set, poll advances it each cycle and emits delta events, snapshot renders the current view. The runtime handles scheduling, history, and error tracking.

#[async_trait]
pub trait Observer: Send + Sync + 'static {
    type Source: Clone + Send + Sync + 'static;
    type State: Send + Sync + 'static;
    type Snapshot: Clone + Send + Sync + 'static;
    type Event: Clone + Send + Sync + 'static;

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError>;
    async fn poll(&self, sources: &[ObservedSource<Self::Source>], state: &mut Self::State)
        -> Result<Vec<Self::Event>, DynError>;
    fn snapshot(&self, state: &Self::State) -> Self::Snapshot;
}

SourceProvider<S> returns the current source set before each cycle, which lets the observed population change mid-run. Use StaticSourceProvider for a fixed set.

#[async_trait]
pub trait SourceProvider<S>: Send + Sync + 'static {
    async fn sources(&self) -> Result<Vec<ObservedSource<S>>, DynError>;
}

SourceProviderFactory<E, S> builds the provider once node clients exist. Any Fn(&E::Deployment, NodeClients<E>) -> Result<BoxedSourceProvider<S>, DynError> closure implements it. All three plug into a scenario through ObservationExtensionFactory<E, O>, which is itself a RuntimeExtensionFactory; see examples/openraft_kv/testing/integration/src/observation.rs for a complete implementation.

Application Composition

AppDeployment<E, P> prepares one reusable application preset, such as a process, child cluster, or composed stack, and returns a typed access or control handle. Framework adapters register managed resource lifetime separately with scenario cleanup. AppHandle is blanket-implemented for any Clone + Send + Sync + 'static type.

#[async_trait]
pub trait AppDeployment<E, P = LocalClusterProvisioner>: Send + 'static
where
    E: Application,
{
    type Handle: AppHandle;
    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

Registered with AppScenarioBuilderExt::with_app, which wraps it in an AppDeploymentFactory (a RuntimeExtensionFactory). Compose children inside deploy via ctx.deploy(...) / ctx.deploy_and_expose(...). See Handle Ownership and Teardown for handle access and cleanup semantics.

Deployment Backends

ClusterProvisioner<E> is the contract every backend implements: turn a ClusterRequest<E> into a provisioned ClusterUnit<E> (clients, control, wait, cleanup). LocalClusterProvisioner, ComposeProvisioner, and K8sClusterProvisioner are the in-repo implementations; the provisioner is selected per app through with_app_using (local is the with_app default).

#[async_trait]
pub trait ClusterProvisioner<E: Application>: Clone + Send + Sync + 'static {
    async fn provision_cluster(
        &self,
        request: ClusterRequest<E>,
    ) -> Result<ClusterUnit<E>, DynError>;
}

NodeControlHandle<E> is the backend-agnostic control surface behind a ClusterHandle’s node control: start_node(_with), stop_node, restart_node(_with), wait_node_ready, node_client, and node_pid. Every method has a default that returns a “not supported” error, so backends implement only what they support. ClusterWaitHandle<E> provides the cluster-wide wait_network_ready operation. Both are combined by ManualClusterHandle<E> in core::runtime::manual, the interface behind ManualCluster.

Local Binary Resolution

BinaryProvider resolves the executable path for a locally spawned node process. Implementations return Ok(None) when valid but unable to resolve, which is how FallbackBinaryProvider chains providers. The default resolve caches per process by cache_key.

pub trait BinaryProvider: Send + Sync {
    fn try_resolve(&self) -> Result<Option<PathBuf>, BinaryProviderError>;
    fn display(&self) -> String;
    fn cache_key(&self) -> String;
    // provided: resolve(), resolve_uncached()
}

Built-in implementations: PathBinaryProvider, EnvBinaryProvider, BuildBinaryProvider, DownloadBinaryProvider, FallbackBinaryProvider. DownloadProcessor post-processes a checksum-verified download (for example, unpacking an archive) into the executable; DownloadProcessorFn adapts a closure with a stable cache_key so changed preparation logic invalidates the cache.

pub trait DownloadProcessor: Send + Sync {
    fn process(&self, artifact: &Path, output: &Path) -> Result<(), DownloadProcessorError>;
    fn cache_key(&self) -> &str;
}

Attaching Sources

IntoExistingCluster converts a value into the typed ExistingCluster descriptor consumed by ClusterRequest::attached. It is implemented for ExistingCluster and &ExistingCluster; implement it for your own environment-selection types to keep attach logic in one place. External endpoints use ExternalNodeSource values directly and pair with Application::external_node_client.

The required extension points depend on the entry pattern: a uniform managed cluster needs Application and the scenario traits, an AppHost stack adds AppDeployment, and attached clusters add the source traits. See Choosing an Entry Pattern. For where each implementation should live, see Framework vs Application Boundaries and the crate-level view in Crate and API Map.

Crate and API Map

This chapter maps which crate owns which concept, what each one exports, and how they depend on each other.

The workspace splits into the app-agnostic core, portable resource contracts, the application-composition layer, deployment backends, and the cfgsync configuration pipeline. Example applications live under examples/ and depend on the framework, never the other way around.

graph BT
    art[cfgsync-artifacts]
    cc[cfgsync-core] --> art
    ca[cfgsync-adapter] --> cc
    ca --> art
    cr[cfgsync-runtime] --> ca
    core[testing-framework-core] --> ca
    container[testing-framework-container] --> core
    local[testing-framework-runner-local] --> core
    compose[testing-framework-runner-compose] --> core
    compose --> container
    k8s[testing-framework-runner-k8s] --> core
    k8s --> cc
    k8s --> art
    app[testing-framework-app] --> core
    app --> container
    app --> local

testing-framework-core

Path: testing-framework/core. The scenario engine and everything app-agnostic: builder, runtime, topology, observation, sources, capabilities. Every other framework crate depends on it.

Key builder entry points: ScenarioBuilder::with_deployment and ::new(provider). The shared fluent methods (with_workload, with_expectation, with_observer, policy setters, …) live on CoreBuilderExt.

testing-framework-container

Path: testing-framework/container. The portable contract for containerized workloads. It owns ContainerServiceSpec, ContainerStackRequest, ContainerStackProvisioner, resolved endpoints, cleanup ownership, and per-container lifecycle handles. It contains no application composition, Docker commands, Compose descriptors, Helm values, Kubernetes resources, or Local-process implementation.

The app crate consumes this contract when an AppDeployment provisions container resources. Compose implements it today; Kubernetes can implement the same contract without either backend depending on the app crate.

testing-framework-app

Path: testing-framework/app. The composition layer for heterogeneous stacks: singleton processes, extra clusters, container stacks, or several applications composed into one system. It consumes Local and container contracts without owning either backend model (see Backend Scope).

Deployment Backends

Each backend implements ClusterProvisioner<E> for its environment trait; the provisioner is passed per app through with_app_using (local is the with_app default).

testing-framework-runner-local (testing-framework/deployers/local) spawns nodes as local processes. Exports LocalClusterProvisioner, LocalCluster, ManualCluster, NodeManager, the LocalDeployerEnv / LocalBinaryApp environment traits with config/port helpers (LocalProcessSpec, LocalNodePorts, build_local_cluster_node_config, …), process primitives (LaunchSpec, NodeEndpoints, ProcessNode), and the whole binary module (BinaryProvider and its implementations). Honors TF_KEEP_LOGS for tempdir retention.

testing-framework-runner-compose (.../compose) renders Docker Compose stacks. Exports ComposeProvisioner (which implements both ClusterProvisioner and ContainerStackProvisioner), ComposeDeployEnv, descriptor builders (ComposeDescriptor, NodeDescriptor), compose lifecycle commands (compose_up, compose_down, dump_compose_logs), and the Docker config-server support used to serve cfgsync artifacts to containers. The cluster and container-stack roles are adapters over one internal managed-project runtime rather than independent Compose implementations.

testing-framework-runner-k8s (.../k8s) installs a Helm release. Exports K8sClusterProvisioner, K8sDeployEnv, ManualCluster (K8s variant), Helm/chart-value infrastructure (HelmInstallSpec, RunnerChartValues, render_binary_config_node_chart_assets, …), and wait/cleanup helpers. Depends directly on cfgsync-core and cfgsync-artifacts for artifact delivery.

cfgsync

cfgsync is the typed pipeline that turns app config into per-node files: app config → registration snapshot → per-node artifact sets → backend rendering. Consumed by the compose and k8s deployers (locally, configs are written straight to disk). See Static Artifacts and cfgsync.

Examples Workspace Layout

Every example app follows the same four-part shape under examples/<app>/:

examples/kvstore/
├── kvstore-node/            # the application binary under test
├── testing/
│   ├── integration/         # crate kvstore-runtime-ext: Application impl,
│   │                        #   local/compose/k8s env impls, observation
│   └── workloads/           # crate kvstore-runtime-workloads: Workloads + Expectations
└── examples/                # crate kvstore-examples: runnable bins

The naming is uniform: <app>-runtime-ext, <app>-runtime-workloads, <app>-examples. nats and redis_streams have no node crate because they run upstream binaries or images. multi_app uses an acceptance-suite layout instead: a job-worker/ binary crate, a fixture/ crate (multi-app-fixture: the stack deployment, handles, workload, and expectation), and an e2e/ crate (multi-app-e2e) whose integration tests drive the fixture. It demonstrates application composition.

Run any example bin with:

cargo run -p kvstore-examples --bin kvstore_basic_convergence

Note: the dependency arrows only ever point from examples toward the framework and from backends toward core. If you find yourself wanting an arrow in the other direction, read Framework vs Application Boundaries. The trait-level view of the same surface is in Public Extension Points.

Framework vs Application Boundaries

This chapter covers the rules that keep the framework application-agnostic and how they are enforced in practice.

Ownership and Design Boundaries explains the ownership split. This chapter covers the concrete rules, enforcement mechanisms, and signs that code is in the wrong layer.

The Rule

Dependencies point in exactly one direction: application repositories depend on framework crates, never the reverse. The framework knows applications only through the traits in Public Extension Points; everything app-specific (node configs, HTTP clients, readiness semantics, observers, workloads) lives in the application’s own integration crates.

The in-repo examples/ workspace models the application side: each app has an integration crate (<app>-runtime-ext) implementing Application and the per-backend environment traits, and a workloads crate (<app>-runtime-workloads) implementing Workload and Expectation. No framework crate names an example app in its Cargo.toml; see the dependency diagram in Crate and API Map.

graph LR
    subgraph Application side
        ext["&lt;app&gt;-runtime-ext<br/>Application, env impls, Observer"]
        wl["&lt;app&gt;-runtime-workloads<br/>Workload, Expectation"]
    end
    subgraph Framework side
        core[testing-framework-core]
        appl[testing-framework-app]
        runners[deployers]
    end
    ext --> core
    ext --> runners
    wl --> core
    appl --> core
    runners --> core

Traits cross the boundary; concrete application types never do.

What belongs where:

Enforcement Mechanisms

The type system. The scenario engine is generic over Application, so core code physically cannot reference your node client or config, because there is no concrete type to name. The app layer goes further: AppHostEnv sets NodeClient = () and its build_node_client returns an error, forcing application clients to travel as typed handles owned by the app side rather than leaking into the environment.

Runtime registration errors. Two rules are enforced with hard errors instead of silent replacement:

• Registering two runtime extension factories that produce the same type fails at prepare time with duplicate runtime extension type registered: .... This is also why a scenario takes exactly one with_app; compose multiple apps inside one root AppDeployment instead.
• Exposing a handle twice under the same type and name fails with app handle is already exposed: ... (AppDeployError::DuplicateHandle).

The boundary check script. scripts/run/check-boundaries.sh guards the adopter side of the line. What it actually does:

• resolves a sibling adopter checkout at ../nomos-node/tests/testing_framework/lb-topology and fails if it is missing;
• greps that crate’s src/ and Cargo.toml for extension-specific identifiers (cfgsync, ComposeDeployEnv, K8sDeployEnv, runner-compose, runner-k8s, DEFAULT_CFGSYNC_PORT, DEFAULT_ASSETS_STACK_DIR) and fails on any hit.

The same rule can be applied to other integration crates: a topology-level crate remains backend-agnostic, so references to a specific deployer or cfgsync internals are treated as violations. Backend names belong in the per-backend environment modules; compare local_env.rs, compose_env.rs, and k8s_env.rs in examples/openraft_kv/testing/integration/src/.

Crate docs as contract. testing-framework-app states the ownership boundary in its crate docs: implement AppDeployment in the application repository, compose children through DeployContext, and let handles own deployed resources. The multi_app README says the same from the other direction: for composed systems, prefer the app-layer shape “instead of building a fake outer cluster or adding app-specific code to TF”.

Signs Your Code Is on the Wrong Side

Symptoms that application code has leaked into the framework:

• A config template, launch flag, or port convention for one specific application sitting in testing-framework/ or cfgsync/.
• A framework crate importing an example (or adopter) crate, or matching on an application name.
• A “generic” helper in core whose only caller is one app and whose parameters mirror that app’s config fields.

Symptoms that framework mechanics are being re-implemented in the application repo:

• Hand-rolled process spawn/kill/teardown code where LocalProcessApp or ClusterApp would do.
• A custom polling loop with history and error tracking that duplicates the observation runtime; implement Observer instead.
• Re-implementing binary resolution, caching, or fallback chains instead of configuring BinaryProvider types.
• A bespoke “wait until cluster healthy” loop instead of readiness closures plus DeploymentPolicy (see Readiness, Retry, and Artifact Preservation).

A framework addition should compile and make sense with a different application plugged in. Application-specific code belongs in the application repository.

Backend Scope of the App Layer

Cluster composition is backend-explicit: local provisioning requires LocalDeployerEnv, and the Compose and Kubernetes provisioners require their respective environment traits — the provisioner passed to with_app_using names the backend. Container composition crosses the backend boundary through ContainerStackProvisioner: application repositories declare explicit container specs from testing-framework-container, while the Compose crate owns Docker realization and control. Kubernetes should implement that same contract rather than adding application-specific containers to TF. Details are in Backend Scope.

Part VII — Operations

These chapters cover running, integrating, and debugging the framework day to day.

• Running the Examples — every runnable binary and what it exercises
• Continuous Integration — how this repository tests itself, and patterns for yours
• Diagnostics and Retained Artifacts — logs, working directories, post-mortems
• Environment Variables — the complete, audited reference
• Troubleshooting — common failures and their causes
• Glossary — terms used throughout the book

Running the Examples

This chapter lists every runnable example target, the exact command to launch it, and what it needs from your machine.

Conventions

All examples are ordinary binaries run with:

cargo run -p <package> --bin <bin>

Every scenario binary is the same app-model program — AppHost::scenario().with_app(...) deployed through AppHostDeployer — and naming encodes the backend provisioner: *_basic_* binaries use the default LocalClusterProvisioner, *_compose_* pass ComposeProvisioner and need a running Docker daemon, and *_k8s_* pass K8sClusterProvisioner and need a reachable Kubernetes cluster context. *_k8s_manual_* binaries construct the k8s ManualCluster directly and drive it imperatively. Compose binaries exit gracefully with a warning when Docker is unavailable (ComposeRunnerError::DockerUnavailable), and the k8s binaries skip when the cluster cannot be reached (ManualClusterError::ClientInit, or an unreachable-API InstallStack).

Logging uses tracing_subscriber with an env filter; set RUST_LOG to adjust verbosity.

Summary