Testing Framework

System-level testing for networked applications from Rust.

The testing framework deploys and controls processes, containers, and clusters. Tests can run several nodes over network connections for a bounded period. Application-specific configuration and clients stay outside the framework, so the runtime can be used with a key-value store, a Raft cluster, a message queue, or a blockchain.

Get Started

Scenarios

A declarative test is represented by a Scenario containing:

• Topology — the system under test (a uniform cluster, a composed application stack, or attached external nodes)
• Workloads — traffic and conditions that exercise the system
• Expectations — success criteria verified after execution
• Duration — the time window for the experiment

flowchart LR
    subgraph SC["Scenario"]
        T["topology<br/><small>the system under test</small>"]:::cl
        W["workloads<br/><small>drive traffic</small>"]:::sc
        EX["expectations<br/><small>verify outcomes</small>"]:::sc
        D["duration<br/><small>the run window</small>"]:::sc
    end
    SC --> RN["Runner<br/><small>deploy · run · evaluate · teardown</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The scenario runtime executes these parts in the same order for each declarative entry pattern. The entry pattern determines how the system is supplied.

Entry Patterns

flowchart LR
    A[Uniform managed cluster]:::cl --> S[Scenario]
    B[AppHost composed stack] --> S
    C[Attached / external nodes]:::cl --> S
    S:::sc --> R[Runner: workloads + expectations]:::sc
    M[ManualCluster] --> I[Imperative orchestration]
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

Three entry patterns use the scenario runtime:

1. Uniform managed cluster — the framework generates configs and launches N identical nodes from a topology. See Part IV.
2. AppHost composed stack — the app layer deploys heterogeneous components (processes, child clusters, in-process services) as one system and exposes typed handles to workloads. See Part II.
3. Attached and external nodes — the scenario targets clusters you already run, or plain URLs. See Existing and External Clusters.

ManualCluster is the imperative alternative. It provides direct start, stop, restart, and readiness operations without the scenario runner, including for step-driven BDD harnesses.

If you are not sure which to use, read Choosing an Entry Pattern.

Provided APIs

Declarative API

• Express tests as topology + workloads + expectations
• Reuse the same definition across local, Compose, and Kubernetes deployers
• Compose stacks from reusable application deployments

Application layer

• Deploy heterogeneous systems as one root AppDeployment
• Typed, named handles connect workloads to components
• Deterministic cleanup, including on partial-deployment failure

Runtime capabilities

• Capability-gated node control: restart nodes from workloads, portably
• Continuous observation: snapshots, history, and event streams of application state
• Telemetry: metrics, logs, and tracing endpoints

Operations

• Binary providers resolve node binaries from paths, env vars, builds, or downloads
• Reproducible deployments via seeds
• Artifact preservation for post-mortem debugging

Quick Example

use testing_framework_app::{AppHost, AppHostLocalDeployer, AppScenarioBuilderExt as _};
use testing_framework_core::scenario::Deployer as _;

let mut scenario = AppHost::scenario()
    .with_app(KvLocalApp::nodes(3))
    .with_workload(KvAppHostConvergence::new(3))
    .build()?;

let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;

This deploys a three-node key-value store cluster, runs a convergence workload against it (including a node restart), and tears everything down. The remaining chapters cover each part of this pattern in detail.

View the example apps

The Example Apps

The repository includes small applications under examples/ that exercise the framework APIs:

Some chapters also link to adopter repositories. The examples listed in this table run from this workspace.

Documentation Structure

Start with the Quickstart.

Quickstart

Run a complete multi-node test in one command.

Prerequisites

• Rust toolchain (the workspace pins its version via rust-toolchain.toml)
• Unix-like system (tested on Linux and macOS)
• For Compose examples: a running Docker daemon
• For Kubernetes examples: a reachable cluster context

No other setup. Example node binaries are resolved automatically; the kvstore example builds its node with Cargo on first run if no prebuilt binary is available.

Your First Test

git clone <this-repository>
cd <this-repository>
cargo run -p kvstore-examples --bin kvstore_app_host_convergence

First run takes a few minutes (builds the framework and the kvstore-node binary).

What happens:

1. AppHost::scenario() builds a scenario around a composed application instead of a managed node topology.
2. with_app(KvLocalApp::nodes(3)) deploys a three-node kvstore cluster as local processes.
3. The convergence workload writes a value, restarts node-0, waits for readiness, and writes again.
4. The runner evaluates the outcome and tears the cluster down.

What you should see:

• Three kvstore-node processes spawn with generated configs in per-run temporary directories
• The workload logs a successful write before and after the restart
• The command exits successfully and removes the temporary directories

The Code Behind It

The binary is short enough to read in full at examples/kvstore/examples/src/bin/app_host_convergence.rs. Its core is:

let mut scenario = AppHost::scenario()
    .with_app(KvLocalApp::nodes(3))
    .with_run_duration(Duration::from_secs(5))
    .with_workload(KvAppHostConvergence::new(3))
    .build()?;

let deployer = AppHostLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The workload reaches the deployed cluster through a typed handle (RunContext is the object every workload receives at run time; see Part III):

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<LocalAppCluster<KvEnv>>()?;

    put_value(&cluster, "before-restart").await?;
    cluster.restart_node("node-0").await?;
    cluster.wait_node_ready("node-0").await?;
    put_value(&cluster, "after-restart").await?;
    Ok(())
}

The same pattern can run in #[tokio::test] functions. The composition acceptance suite does this:

cargo test -p multi-app-e2e

It uses a reusable fixture crate for the stack, workload, and expectation, then drives them from ordinary integration tests.

Where to Go Next

flowchart LR
    WL["workload<br/><small>enqueue 10 jobs</small>"]:::sc --> Q["queue cluster<br/><small>2 nodes</small>"]:::cl
    Q --> W["worker<br/><small>one process</small>"]:::pr
    W --> R["result store<br/><small>2 nodes</small>"]:::cl
    R --> EX["expectation<br/><small>10 results visible</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

let mut scenario = AppHost::scenario()                // ①
    .with_app(JobStackApp::new())                     // ②
    .with_run_duration(Duration::from_secs(10))       // ③
    .with_workload(EnqueueJobs::new(10))              // ④
    .with_expectation(AllJobsCompleted::new(10))      // ⑤
    .build()?;

let runner = AppHostLocalDeployer::default()
    .deploy(&scenario)                                // ⑥
    .await?;

runner.run(&mut scenario).await?;                     // ⑦

QueueScenario::nodes(5)
    .produce(400).rate_per_sec(40).done()
    .restart_nodes_randomly().every_secs(5, 15).done()
    .partition(["node-0", "node-1"], ["node-2", "node-3", "node-4"]).hold_secs(20).done()
    .expect_converged(400).within_secs(60)
    .run_secs(120)
    .await?;

flowchart LR
    WL["produce<br/><small>400 jobs at 40/s</small>"]:::sc --> A
    subgraph A["partition group A"]
        N0["node-0"]:::cl
        N1["node-1"]:::cl
    end
    subgraph B["partition group B"]
        N2["node-2"]:::cl
        N3["node-3"]:::cl
        N4["node-4"]:::cl
    end
    A -. "✂ split 20s, then heal" .- B
    RR["⚡ random restarts<br/><small>every 5–15s</small>"]:::pr -.-> A
    RR -.-> B
    B --> EX["expect<br/><small>all 5 nodes converge at 400</small>"]:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

let mut scenario = QueueScenarioBuilder::deployment_with(|_| QueueTopology::new(5))
    .enable_node_control()                                //  restarts allowed
    .with_network_control()                               //  partitions allowed
    .with_workload(
        QueueProduceWorkload::new()                       //  steady traffic
            .operations(400)
            .rate_per_sec(40)
            .payload_prefix("soak"),
    )
    .with_workload(RandomRestartWorkload::new(            //  random node restarts
        Duration::from_secs(5),
        Duration::from_secs(15),
        Duration::from_secs(10),
    ))
    .with_workload(NetworkPartitionWorkload::new(         //  split, hold, heal
        NetworkPartitionSpec::new(vec![
            vec!["node-0", "node-1"],
            vec!["node-2", "node-3", "node-4"],
        ]),
        Duration::from_secs(20),
        Duration::from_secs(20),
    ))
    .with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
    .with_run_duration(Duration::from_secs(120))
    .build()?;

let runner = QueueLocalDeployer::default().deploy(&scenario).await?;

runner.run(&mut scenario).await?;

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone;   // cluster shape
    type NodeClient: Clone + Send + Sync;            // how tests reach a node
    type NodeConfig: Clone + Send + Sync;            // per-node config type
}

flowchart TD
    U["Uniform cluster<br/><small>N identical nodes</small>"]:::cl --> S["Scenario"]:::sc
    A["Composed stack<br/><small>the job stack — line ②</small>"]:::sc --> S
    X["Attached / external<br/><small>clusters you already run</small>"]:::cl --> S
    S --> R["Runner<br/><small>one lifecycle for all three</small>"]:::sc
    M["ManualCluster<br/><small>managed nodes, you drive</small>"] -.->|bypasses the runner| C["step-by-step node control"]
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
    let queue = ctx
        .deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes))     // ①
        .await?;

    let results = ctx
        .deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
        .await?;

    let queue_url = queue.first_client().ok_or("queue cluster has no clients")?.base_url().clone();
    let results_url = results.first_client().ok_or("result store has no clients")?.base_url().clone();

    let worker = ctx
        .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))   // ②
        .await?;

    let stack = JobStackHandle { queue, results, worker };              // ③
    ctx.expose(stack.clone())?;

    Ok(stack)
}

struct JobStackHandle {
    queue: LocalAppCluster<QueueEnv>,
    results: LocalAppCluster<KvEnv>,
    worker: LocalProcessHandle<WorkerClient>,
}

let launch = LaunchSpec {
    binary: worker_binary_provider().resolve()?,          // section 9
    args: vec!["--queue-url".to_owned(), queue_url.to_string(), /* … */],
    ..LaunchSpec::default()
};

let process = LocalProcessApp::new("job-worker", launch, endpoints, client)
    .with_readiness(|_, client| async move { client.wait_ready().await });

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))             // register activity
    .with_expectation(AllJobsCompleted::new(10))     // register the check
    .build()?;

let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;

runner.run(&mut scenario).await?;                    // TF invokes both

#[async_trait]
impl Workload<AppHostEnv> for EnqueueJobs {
    fn name(&self) -> &str {
        "enqueue_jobs"
    }

    async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let stack = ctx.require_app::<JobStackHandle>()?;
        let queue = stack
            .queue()
            .first_client()
            .ok_or("queue cluster has no clients")?;

        for index in 0..self.count {
            let response: EnqueueResponse = queue
                .post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
                .await?;

            if !response.accepted {
                return Err(format!("queue rejected job {index}").into());
            }
        }

        Ok(())
    }
}

#[async_trait]
impl Expectation<AppHostEnv> for AllJobsCompleted {
    fn name(&self) -> &str {
        "all_jobs_completed"
    }

    async fn evaluate(&mut self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
        let stack = ctx.require_app::<JobStackHandle>()?;
        let clients = stack.results().clients();
        let deadline = Instant::now() + self.timeout;

        while Instant::now() < deadline {
            if all_results_are_visible(&clients, self.count).await? {
                if !stack.worker().is_running().await {
                    return Err("job worker stopped before evaluation".into());
                }

                return Ok(());
            }

            tokio::time::sleep(Duration::from_millis(200)).await;
        }

        Err(format!("job results did not converge within {:?}", self.timeout).into())
    }
}

flowchart TB
    R["runner sequences the test"]:::driver --> SETUP["system setup"]:::setup
    U["your Rust code / BDD steps<br/>sequence the test"]:::driver --> SETUP

    SETUP --> C["one uniform cluster"]:::shape
    SETUP --> S["composed stack"]:::shape
    S --> CC["uniform child clusters"]:::shape
    S --> P["standalone processes"]:::process

    C --> N["node bring-up<br/><small>topology → ports + peers → config → binary → start → readiness</small>"]:::engine
    CC --> N
    P --> SP["process bring-up<br/><small>launch settings → binary → start → readiness</small>"]:::engine

    N --> H["running resources<br/><small>clients · lifecycle control · reverse cleanup</small>"]:::runtime
    SP --> H

    classDef driver stroke:#9b6dd6,stroke-width:2.5px;
    classDef setup stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef shape stroke:#4a90d9,stroke-width:2.5px;
    classDef process stroke:#e08a3c,stroke-width:2.5px;
    classDef engine stroke:#777,stroke-width:2px;
    classDef runtime stroke:#4caf7d,stroke-width:2.5px;

#[tokio::test]
async fn drives_queue_cluster_without_a_scenario() -> Result<(), DynError> {
    let cluster = ManualCluster::<QueueEnv>::from_topology(QueueTopology::new(2));

    let node0 = cluster.start_node("node-0").await?.client;
    let node1 = cluster.start_node("node-1").await?.client;
    cluster.wait_network_ready().await?;

    enqueue(&node0, "manual-job").await?;
    wait_for_queue_len(&[node0, node1], 1).await?;

    cluster.restart_node("node-1").await?;
    cluster.wait_node_ready("node-1").await?;

    let restarted = cluster
        .node_client("node-1")
        .ok_or("node-1 client missing after restart")?;
    wait_for_queue_len(&[restarted], 1).await?;

    Ok(())
}

let mut deployment =
    DeployContext::<AppHostEnv>::new(AppHostTopology, NodeClients::default());
let stack = deployment.deploy(JobStackApp::new()).await?;

assert_eq!(stack.queue().node_count(), 2);
assert_eq!(stack.results().node_count(), 2);

let queue = stack.queue().first_client().ok_or("queue has no clients")?;
let results = stack.results().clients();
let worker = stack.worker().clone();

worker.restart().await?;
enqueue(&queue, "imperative-job").await?;
wait_for_completed_result(&results, "imperative-job").await?;

drop(deployment);                         // reverse cleanup for the whole stack
assert!(!worker.is_running().await);

flowchart TB
    T["1 · topology<br/><small>the test asks for three queue nodes</small>"]:::input
    V["2 · framework prepares node 1<br/><small>identity · reserved port · peer addresses</small>"]:::framework
    C["3 · application builds queue configuration<br/><small>node id · HTTP port · peers · sync interval</small>"]:::app
    B["4 · backend launches the node<br/><small>write config file · resolve binary · pass args and environment · start</small>"]:::backend
    H["5 · ready running resource<br/><small>typed client · lifecycle control · registered cleanup</small>"]:::runtime

    T --> V --> C --> B --> H

    POL["deployment policy<br/><small>readiness · retry · retained artifacts</small>"]:::policy -. "governs launch" .-> B
    POL -. "gates access" .-> H

    classDef input stroke:#777,stroke-width:2px;
    classDef framework stroke:#4a90d9,stroke-width:2.5px;
    classDef app stroke:#9b6dd6,stroke-width:2.5px;
    classDef backend stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef runtime stroke:#4caf7d,stroke-width:2.5px;
    classDef policy stroke:#c89b3c,stroke-width:2.5px;

fn build_cluster_node_config(
    node: &ClusterNodeView,
    peers: &[ClusterPeerView],
) -> Result<QueueNodeConfig, Error> {
    Ok(QueueNodeConfig {
        node_id: node.index() as u64,
        http_port: node.network_port(),
        peers: peers
            .iter()
            .map(|peer| QueuePeerInfo {
                node_id: peer.index() as u64,
                http_address: peer.authority(),
            })
            .collect(),
        sync_interval_ms: 500,
    })
}

fn local_process_spec() -> LocalProcessSpec {
    LocalProcessSpec::new("QUEUE_NODE_BIN")
        .with_binary_provider(queue_binary_provider())
        .with_rust_log("queue_node=info")
}

fn render_local_config(config: &QueueNodeConfig) -> Result<Vec<u8>, DynError> {
    yaml_node_config(config)
}

fn http_api_port(config: &QueueNodeConfig) -> u16 {
    config.http_port
}

flowchart TB
    C["typed per-node configuration"]:::app --> A["rendered per-node artifacts<br/><small>config file + any additional files</small>"]:::artifact
    A --> L["local backend<br/><small>write directly into the node working directory</small>"]:::local
    A --> S["container backends<br/><small>serve artifacts through cfgsync</small>"]:::container
    S --> F["cfgsync client in each container<br/><small>register · fetch · write files</small>"]:::container
    L --> N["start node binary"]:::process
    F --> N

    classDef app stroke:#9b6dd6,stroke-width:2.5px;
    classDef artifact stroke:#777,stroke-width:2px,stroke-dasharray:4 3;
    classDef local stroke:#4a90d9,stroke-width:2.5px;
    classDef container stroke:#c89b3c,stroke-width:2.5px;
    classDef process stroke:#e08a3c,stroke-width:2.5px;

let node1 = cluster
    .start_node_with(
        "node-1",
        StartNodeOptions::<QueueEnv>::default().create_patch(|mut config| {
            config.sync_interval_ms = 50;
            Ok(config)
        }),
    )
    .await?
    .client;

let policy = DeploymentPolicy {
    readiness_enabled: true,
    readiness_requirement: HttpReadinessRequirement::AtLeast(2),
    retry_policy: Some(RetryPolicy::new(
        5,
        Duration::from_millis(500),
        Duration::from_secs(5),
    )),
    cleanup_policy: CleanupPolicy::new(true),
    ..DeploymentPolicy::default()
};

let cluster  = ctx.deploy_cluster(ClusterRequest::managed(deployment)).await?;
let attached = ctx.deploy_cluster(ClusterRequest::attached(existing)).await?;
let external = ctx.deploy_cluster(ClusterRequest::external(endpoints)).await?;

FallbackBinaryProvider::new([
    Arc::new(EnvBinaryProvider::new("MULTI_APP_JOB_WORKER_BIN")),
    Arc::new(BuildBinaryProvider {
        command: BuildCommand::new("cargo")
            .with_args(["build", "-p", "multi-app-job-worker", "--bin", "multi-app-job-worker"]),
        output_path: "target/debug/multi-app-job-worker".into(),
        working_dir: Some(workspace),
        lock_dir: None,
    }),
])

Part I — Mental Model

This part defines the main framework abstractions.

This part explains what the framework’s core types mean, how a scenario executes from build to teardown, and how to pick the right entry pattern for a given test before writing any code.

• Application, AppDeployment, and Environments — the three roles a “thing under test” can play
• Scenario Model and Lifecycle — what a scenario is and every phase it passes through
• Choosing an Entry Pattern — uniform cluster, composed stack, attached nodes, or manual control
• Ownership and Design Boundaries — what the framework owns versus what your application repository owns

Application, AppDeployment, and Environments

This chapter distinguishes the Application trait, the AppDeployment trait, and the concrete environment types that implement Application.

The Application Trait

Application is the contract between the scenario engine and whatever system you are testing. It bundles the backend-specific types the engine needs, without the engine ever knowing what your application does:

use testing_framework_core::scenario::Application;

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str; // default: "/"
}

The associated types and methods are:

• Deployment: the topology descriptor, i.e. how many nodes exist and how they relate.
• NodeClient: the typed client workloads use to talk to one node.
• NodeConfig: the per-node configuration your binary consumes.
• Client constructors: build_node_client turns deployer-provided NodeAccess into a client; external_node_client does the same for nodes the framework did not start. Both return an “unsupported” error by default. An environment must implement the operations it supports (Ownership and Design Boundaries).
• node_readiness_path: the HTTP path deployers probe during readiness gating.

An implementation of Application is called an environment. Everything generic in the framework (ScenarioBuilder<E>, Workload<E>, Expectation<E>, RunContext<E>) is parameterized over one.

Source: testing-framework/core/src/env.rs.

The AppDeployment Trait

Application describes a uniform node population. A system containing a cluster plus another process, or several different clusters, is represented through AppDeployment in testing-framework-app:

use testing_framework_app::{AppDeployment, AppHandle, DeployContext};

pub trait AppDeployment<E: Application, P = LocalClusterProvisioner>: Send + 'static {
    type Handle: AppHandle;

    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

An AppDeployment is a deployable unit: it consumes its description, prepares whatever it represents, and returns a typed runtime handle. AppHandle is a blanket implementation, so any Clone + Send + Sync + 'static type qualifies. The handle provides access and control; managed resources acquired through framework adapters are owned separately by scenario cleanup.

Deployments compose: inside deploy, the DeployContext lets a parent deployment call ctx.deploy(child) or ctx.deploy_and_expose(child), then ctx.expose(handle) to publish typed handles to workloads. See AppDeployment and DeployContext for the full context API.

An AppDeployment registered with .with_app(...) runs during scenario preparation. It participates in the scenario lifecycle; it does not replace that lifecycle.

Source: testing-framework/app/src/deployment.rs.

Concrete Environments

AppHostEnv: an environment without outer nodes

AppHostEnv is an environment with no outer nodes at all. Its topology, AppHostTopology, reports a node count of zero; its NodeClient and NodeConfig are both (); asking it for a node client is an error. It exists so that a scenario can be composed entirely from application deployments:

use testing_framework_app::{AppHost, AppScenarioBuilderExt};

let builder = AppHost::scenario()      // ScenarioBuilder<AppHostEnv>, zero nodes
    .with_app(KvLocalApp::nodes(3));   // apps provide all processes

The system is supplied by with_app deployments, and workloads access it through typed handles instead of outer node clients. See AppHost and with_app.

Source: testing-framework/app/src/host.rs.

KvEnv: a uniform node environment

The kvstore example shows a full environment for a real binary:

pub struct KvEnv;

impl Application for KvEnv {
    type Deployment = KvTopology;        // ClusterTopology
    type NodeClient = KvHttpClient;
    type NodeConfig = KvNodeConfig;

    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError> {
        Ok(KvHttpClient::new(access.api_base_url()?))
    }

    fn node_readiness_path() -> &'static str {
        "/health/ready"
    }
}

KvEnv additionally implements LocalBinaryApp (in examples/kvstore/testing/integration/src/local_env.rs) to tell the local deployer which binary to run, how to render per-node configs, and which port serves the HTTP API. The same environment type also backs AppDeployment presets like KvLocalApp, whose handle is a whole child cluster. Implementing Application walks through this in detail.

Source: examples/kvstore/testing/integration/src/app.rs.

How the Three Relate

graph TD
    SB["ScenarioBuilder&lt;E&gt;"] -->|"E: Application"| APP["Application<br/>(env contract)"]
    KV["KvEnv"] -.->|implements| APP
    AH["AppHostEnv<br/>(zero-node env)"] -.->|implements| APP
    AD["AppDeployment&lt;E, P&gt;"] -->|"deploys via"| DC["DeployContext&lt;E, P&gt;"]
    AD -->|returns| H["typed Handle"]
    SB -->|".with_app(...)"| AD
    SB:::sc
    H:::hd
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

Application and AppDeployment are not alternatives. Every scenario has one environment type E, and app deployments are registered inside that scenario. A uniform kvstore cluster uses KvEnv directly; a heterogeneous stack uses AppHostEnv and supplies its components as app deployments.

Where to Go Next

• Scenario Model and Lifecycle: what a scenario is and how it runs.
• Choosing an Entry Pattern: which combination of these pieces fits your system.
• Part II — Composing Applications: the app layer in depth.
• Part IV — Uniform Clusters and Configuration: implementing an environment for your own binary.

Scenario Model and Lifecycle

A scenario records a topology, workloads, expectations, runtime settings, and deployment policy. The runner executes the phases described below.

What a Scenario Is

You assemble a scenario with ScenarioBuilder<E> and hand it to a deployer. The essential ingredients:

Because the whole plan is declared up front, build() can validate it and fail before any process is spawned.

Workloads implement Workload<E> (name(), init(...), async start(&self, ctx)); expectations implement Expectation<E> (start_capture, optional check_during_capture, evaluate). A workload can also contribute its own expectations; with_workload collects them automatically. Both receive the shared RunContext<E>, which carries the deployment descriptor, node clients, telemetry, and typed runtime extensions. See Workloads and Concurrency and Expectations and Evaluation.

The Lifecycle

flowchart TD
    B["build()"] --> D["deployer.deploy(&scenario)"]
    D --> RG["spawn + readiness gating (retry per policy)"]
    RG --> PX["prepare runtime extensions (with_app runs here)"]
    PX --> RUN["runner.run(&mut scenario)"]
    RUN --> W["workloads start concurrently"]
    W --> CD["cooldown window"]
    CD --> EV["evaluate all expectations (aggregate failures)"]
    EV --> H["RunHandle"]
    H --> T["drop → cleanup guards"]
    W -- "failure" --> T
    EV -- "failure" --> T
    RUN:::sc
    W:::sc
    EV:::sc
    H:::hd
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

1. Build

build() finalizes the plan. It resolves the deployment from the topology provider (honoring with_deployment_seed), validates the source configuration (for example, external-only scenarios must declare at least one external node, and node control is rejected for uncontrolled external clusters), and calls init on every workload and expectation. Failures surface as ScenarioBuildError before anything is deployed.

Note: build() enforces a minimum run duration of 10 seconds and defaults the expectation cooldown to 10 seconds when you have not set one.

2. Deploy

deployer.deploy(&scenario) provisions the environment and returns a Runner<E>. For the local deployer this means spawning node processes, then readiness gating: each node’s readiness probe (HTTP path or plain TCP, per the app) is retried until the policy’s readiness requirement holds, with retry and backoff per DeploymentPolicy. Only after the cluster is ready are runtime extensions (typed services prepared once per run and handed to workloads, see Runtime Extensions) prepared; with_app deployments deploy at this point. Registering two extensions of the same type fails here with a “duplicate runtime extension type registered” error. Failure-path cleanup already applies at this stage through ownership: when deployment errors partway, partially deployed app resources are released as their handles drop, and spawned node processes stop when their process handles drop.

3. Run: workloads

runner.run(&mut scenario) first calls start_capture on every expectation, then spawns all workloads concurrently, each in its own task. Workload panics are caught and converted into workload errors instead of aborting the process. The run window lasts for the configured duration, during which the runner also ticks check_during_capture on every expectation once per second, so an expectation can fail during the run instead of waiting for final evaluation.

A workload returning early with Ok(()) is fine; the window keeps running while other workloads are still active. The duration is a maximum: once every workload has finished, the window ends early and cooldown begins. A workload error ends the run immediately with ScenarioError::Workload.

4. Cooldown and settle

When the duration elapses, workloads are not cut off abruptly. The runner keeps the run alive through a cooldown window derived from with_expectation_cooldown; clusters whose lifecycle the framework owns get a 30-second minimum so restarted nodes and runtime extensions observe stabilized state. Remaining workload tasks are then drained, and a short settle wait (at least 2 seconds when a cooldown or node control is in play) runs before evaluation.

5. Evaluation

Every expectation’s evaluate runs, including after another expectation fails. Failures are aggregated into one ScenarioError::Expectations report with one line per failed expectation.

6. Teardown

A successful run returns a RunHandle<E>. Teardown is guard-based. When the handle drops, its CleanupGuard chain runs, stopping node processes, aborting extension tasks, and executing app cleanup stacks (see Handle Ownership and Teardown). The same guards run on the failure path: any step that errors inside run triggers immediate cleanup before the error is returned, so failed runs do not leak managed processes or temp directories.

let mut scenario = KvScenarioBuilder::deployment_with(|t| t)
    .with_run_duration(Duration::from_secs(30))
    .with_expectation_cooldown(Duration::from_secs(5))
    .with_workload(KvWriteWorkload::new().operations(300))
    .with_expectation(KvConverges::new("demo", 30))
    .build()?;

let deployer = KvLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
let _handle = runner.run(&mut scenario).await?;
// dropping _handle tears the cluster down

Source: testing-framework/core/src/scenario/runtime/runner.rs and runtime/context.rs.

Errors by Phase

Where to Go Next

• Application, AppDeployment, and Environments: the type parameter behind ScenarioBuilder<E>.
• Choosing an Entry Pattern: the ways to reach this one lifecycle.
• Readiness, Retry, and Artifact Preservation: tuning the deploy phase.
• Part III — Scenario Runtime: workloads, expectations, and capabilities in depth.

Choosing an Entry Pattern

The framework supports three scenario-based entry patterns and one imperative entry pattern. This chapter compares them.

The Four Patterns

1. Uniform managed cluster. Your system is N identical nodes of one application. Implement Application (and the deployer-specific traits), describe a topology, and build a ScenarioBuilder<E> over it. The framework spawns, gates, and tears down every node.

let mut scenario = KvScenarioBuilder::deployment_with(|t| t)   // 3-node default topology
    .with_run_duration(Duration::from_secs(30))
    .with_workload(KvWriteWorkload::new().operations(300))
    .with_expectation(KvConverges::new("demo", 30))
    .build()?;

let runner = KvLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;

(The shipped kvstore_basic_convergence binary wraps the same topology in the with_existing_kvstore_app convenience preset; that hybrid is covered in AppHost and with_app.)

2. Composed application stack. Your system is heterogeneous: several clusters, singleton processes, or both. Start from AppHost::scenario() (a zero-node ScenarioBuilder<AppHostEnv>) and register one root AppDeployment with .with_app(...). The deployment composes children through DeployContext and exposes typed handles that workloads retrieve with AppRunContextExt.

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())              // queue cluster + worker + result store
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))
    .with_expectation(AllJobsCompleted::new(10))
    .build()?;

let runner = AppHostLocalDeployer::default().deploy(&scenario).await?;
runner.run(&mut scenario).await?;

A scenario accepts one with_app registration. A second registration fails at prepare time with a duplicate-runtime-extension error; compose multiple apps inside one root deployment instead (Composing Heterogeneous Stacks).

3. Attached and external nodes. The system already runs somewhere else: a staging network, a long-lived cluster, another team’s deployment. You plug it in as a source instead of deploying it: with_existing_cluster(...) / with_existing_cluster_from(...) attach a cluster description, with_external_node(...) / with_external_nodes(...) add endpoint-only nodes, and with_external_only_nodes(...) declares a scenario with no framework-managed nodes at all. Application::external_node_client turns each ExternalNodeSource into a typed client. Workloads and expectations are unchanged. See Existing and External Clusters.

4. ManualCluster: imperative control. Your code decides when nodes start, stop, and restart, step by step. ManualCluster::from_topology(descriptors) (or ProcessDeployer::manual_cluster_from_descriptors) gives you start_node, start_node_with(StartNodeOptions), stop_node, restart_node, wait_network_ready, wait_node_ready, and node_client, but no workloads, no expectations, no runner. See ManualCluster: Imperative Node Control.

Note: needing to restart nodes does not push you to ManualCluster. Declarative scenarios gain restart-capable workloads via with_node_control() on the builder (Scenario Capabilities), and app-layer child clusters expose restart_node on their handles.

One Runtime, Three Declarative Patterns

flowchart TD
    U["Uniform cluster<br/>ScenarioBuilder::with_deployment"] --> S["Scenario"]
    A["Composed stack<br/>AppHost::scenario().with_app(...)"] --> S
    X["Attached / external<br/>with_existing_cluster,<br/>with_external_nodes"] --> S
    S --> R["Deployer::deploy → Runner::run<br/>(one lifecycle, see Scenario Model)"]
    M["ManualCluster<br/>managed nodes, you drive"] -.->|"bypasses the runner"| C["imperative node control"]
    S:::sc
    R:::sc
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

All three declarative patterns produce a Scenario and use the same lifecycle, so workloads and expectations can be reused across them when their required clients and capabilities are available. ManualCluster uses the node-startup implementation without the scenario runtime. Its nodes remain framework-managed while test code controls the sequence.

Decision Table

Choosing by Example

“Three kvstore nodes, write traffic, convergence check.” Uniform managed cluster. KvEnv already models the node; the framework owns the whole population. This is cargo run -p kvstore-examples --bin kvstore_basic_convergence.

“A queue cluster, a worker process, and a result-store cluster forming one pipeline.” Composed stack. One root AppDeployment deploys both clusters, wires the worker to them by URL, and exposes each handle plus a stack handle; a workload enqueues jobs and an expectation verifies the results. The multi-app-e2e acceptance test covers this shape; run it with cargo test -p multi-app-e2e.

“Run our smoke workload against the live staging network.” Attach. There is nothing to deploy: declare the endpoints with with_external_only_nodes, let external_node_client build clients, and keep the exact same workloads and expectations you use locally.

“A Gherkin suite where each step starts or kills a node.” ManualCluster. The BDD runner owns sequencing, and its steps call start_node_with, stop_node, and wait_node_ready directly.

External example: logos-blockchain’s cucumber suite is a real example of the fourth pattern: Gherkin steps drive ManualCluster for dependency-ordered starts, restarts, and snapshot/restore flows, all in its own repository.

Where to Go Next

• Scenario Model and Lifecycle: the runtime every declarative pattern converges on.
• Application, AppDeployment, and Environments: the types behind patterns 1 and 2.
• Ownership and Design Boundaries: what stays yours regardless of pattern.

Ownership and Design Boundaries

This chapter lists the responsibilities of the framework and of an application repository.

The Boundary

The scenario engine never names a concrete application. Its only coupling point is the Application trait: a bundle of associated types (Deployment, NodeClient, NodeConfig) that the engine plumbs around generically. Everything that knows what your system is (its binary, its config format, its client, its notion of “healthy”) lives on your side of that trait.

What the Framework Owns

Process lifetime and working directories. Deployers spawn node processes into per-run working directories, track PIDs, and stop everything on teardown. Artifact retention is policy (CleanupPolicy::preserve_artifacts), not something scenarios hand-roll.

Cleanup. Teardown is guard-based and automatic: cleanup guards chain and run in reverse registration order when the RunHandle drops, and the same guards run on the failure path. App-layer adapters register managed resources in a LIFO cleanup stack so dependants stop before dependencies, independently of exposed handle clones (Handle Ownership and Teardown).

Topology mechanics. Port allocation, peer wiring, node naming, and readiness gating with retry are all generic over E: Application. The engine asks your environment what to render and probe, never why.

Handle storage and lookup. DeployContext collects typed handles during preparation; AppRuntime carries them into the run; AppRunContextExt returns clones to workloads. Duplicate exposure of a type/name pair is an error, never a silent replacement.

What the Application Repository Owns

The kvstore example is the template. Its integration crate supplies, in its own repository:

• The binary and how to get it: a FallbackBinaryProvider chain that uses KVSTORE_NODE_BIN if set and otherwise builds kvstore-node with cargo (Binary Providers).
• Config: KvNodeConfig, built per node from the framework’s port/peer views and rendered to YAML.
• Client: KvHttpClient, constructed in Application::build_node_client.
• Readiness: node_readiness_path() returning /health/ready.
• Domain handles and presets: KvStoreCluster, KvLocalApp, KvExistingClusterApp.
• Scenarios that mean something: write workloads, convergence expectations, runnable bins.

// Application side: domain knowledge, no orchestration.
fn node_readiness_path() -> &'static str {
    "/health/ready"
}

// Framework side: orchestration, no domain knowledge.
// It only ever sees E::NodeClient, E::NodeConfig, E::Deployment.

Sources: examples/kvstore/testing/integration/src/{app,local_env}.rs, testing-framework/app/src/lib.rs.

How the Boundary Is Enforced

Unsupported defaults. Application::build_node_client and external_node_client return an “unsupported” error by default. Capabilities are available only when the environment implements them.

Generic application types. There is no global list of known applications or framework config file naming their binaries. testing-framework-core compiles against E: Application, so it does not depend on adopter types or their dependencies. The same runtime can therefore be instantiated with kvstore, openraft_kv, nats, or an application from another repository.

Application-owned composition. Application repositories implement AppDeployment, compose children through DeployContext, and expose typed handles. The framework supplies the context, registry, and lifecycle without defining the application stack.

CI boundary check. scripts/run/check-boundaries.sh checks an application-side topology crate for framework-extension symbols (cfgsync, ComposeDeployEnv, K8sDeployEnv, runner-compose, runner-k8s). This detects backend dependencies in topology code. The compiler enforces the reverse direction because core crates do not reference concrete application types.

External example: the current boundary script targets logos-blockchain’s lb-topology crate (in its own checkout), which keeps that adopter’s topology code local/topology-focused. The pattern generalizes: point the same grep at your own integration crates.

Application-specific config formats and startup rules belong in the environment implementation or an AppDeployment, not in framework crates.

Where to Go Next

• Application, AppDeployment, and Environments: the trait that defines the boundary.
• Implementing Application: building your side of it.
• Framework vs Application Boundaries: the reference treatment in Part VI.
• Public Extension Points: the sanctioned ways to extend the framework itself.

Part II — Composing Applications

The app layer deploys heterogeneous systems as one unit and exposes typed handles to workloads.

Use this entry pattern when the system under test is not a single uniform cluster. A root AppDeployment deploys children such as processes, uniform child clusters, and in-process services. It exposes their handles, while the scenario runtime schedules test behavior and cleanup.

• AppHost and with_app — hosting a composed app inside a scenario
• AppDeployment and DeployContext — the deployment contract and its context
• Handle Ownership and Teardown — typed access, managed lifetime, and reverse cleanup
• One Binary: LocalProcessApp — the smallest building block
• Uniform Child Clusters: LocalAppCluster — a managed cluster as one component
• Composing Heterogeneous Stacks — the root-app pattern, end to end
• Backend Scope — what the app layer supports today

AppHost and with_app

AppHost creates a scenario whose system under test is supplied by application deployments instead of an outer managed node topology.

The core scenario engine models one Application and a uniform cluster of its nodes. For a composed stack containing a binary, an additional cluster, or several applications, start from AppHost::scenario() and register the stack with .with_app(...). Workloads, expectations, run duration, and teardown follow the lifecycle in Scenario Model and Lifecycle.

The Zero-Node Scenario

AppHost::scenario() returns a ScenarioBuilder<AppHostEnv> seeded with AppHostTopology:

Because the outer topology is empty, app deployments create the processes and clusters used by the run.

use testing_framework_app::{AppHost, AppHostLocalDeployer, AppScenarioBuilderExt};
use testing_framework_core::scenario::Deployer;

let mut scenario = AppHost::scenario()
    .with_app(KvLocalApp::nodes(3))
    .with_run_duration(Duration::from_secs(5))
    .with_workload(KvAppHostConvergence::new(3))
    .build()?;

let deployer = AppHostLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The runnable kvstore_app_host_convergence binary uses this structure:

cargo run -p kvstore-examples --bin kvstore_app_host_convergence

How with_app Runs

AppScenarioBuilderExt::with_app(app) wraps your AppDeployment in an AppDeploymentFactory and registers it as a runtime extension factory, the same lifecycle hook covered in Runtime Extensions. Going through the extension mechanism ties managed deployment cleanup to the scenario lifetime and makes exposed handles available during the run.

flowchart LR
    B["with_app(app)"] --> F[AppDeploymentFactory]
    F -->|prepare| C[DeployContext]
    C -->|"deploy(root app)"| H["handles + cleanup"]
    H --> R[AppRuntime extension]
    R -->|require_app| W[Workloads]
    H:::hd
    R:::hd
    W:::sc
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

During scenario preparation the factory:

1. Clones your app (this is why the factory requires Clone) and builds a fresh DeployContext.
2. Runs the root deployment’s deploy, which may deploy and expose child apps.
3. Auto-exposes the returned root handle if the deployment did not expose one of that type itself (!ctx.contains::<A::Handle>()).
4. Transfers the handle registry and cleanup stack into an AppRuntime extension.

If any step fails, the partially built context is dropped and every resource deployed so far is released (see Handle Ownership and Teardown).

A scenario accepts one with_app registration. Every AppDeploymentFactory produces the same extension type (AppRuntime), and the runtime rejects duplicate extension types. A second registration fails during preparation with duplicate runtime extension type registered: AppRuntime. Compose several applications inside one root AppDeployment and expose the child handles from there, as shown in Composing Heterogeneous Stacks.

with_app Outside AppHost

with_app is defined for every scenario builder, not only AppHostScenarioBuilder. On a regular uniform-cluster scenario, an “existing cluster” preset can wrap the outer scenario’s deployment and node clients in a typed handle without deploying another resource. The OpenRaft example uses this pattern:

// examples/openraft_kv/testing/integration/src/scenario.rs
fn with_existing_openraft_kv_app(app: OpenRaftKvExistingClusterApp) -> Self {
    OpenRaftKvScenarioBuilder::with_deployment(app.topology())
        .with_app(app)
        .with_cluster_observer()
}

Here the scenario still manages a uniform OpenRaft cluster, and the app layer just gives workloads a typed OpenRaftKvCluster handle over it.

Retrieving Handles in Workloads

Workloads never see the deploy context. They retrieve exposed handles through AppRunContextExt, implemented on RunContext<E>:

use testing_framework_app::AppRunContextExt;

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<LocalAppCluster<KvEnv>>()?;
    cluster.restart_node("node-0").await?;
    cluster.wait_node_ready("node-0").await?;
    Ok(())
}

Workloads normally use the require_* variants so that a missing handle produces a typed error containing the requested handle type.

Every retrieval clones the handle. Handles are normally small access values backed by Arc; scenario cleanup still determines managed resource lifetime.

Where to Go Next

• AppDeployment and DeployContext: implementing the deployment itself.
• One Binary: LocalProcessApp and Uniform Child Clusters: LocalAppCluster: the two built-in building blocks.
• Backend Scope: why AppHost scenarios run on the local deployer today.

AppDeployment and DeployContext

Application repositories implement AppDeployment for deployable components. The implementation uses DeployContext to deploy children, expose handles, and register managed resources with scenario cleanup.

The framework runs deployments, stores handles, and performs teardown without defining application binaries or clients. The application crate decides which components start and which typed handles workloads receive.

The Trait Contract

#[async_trait]
pub trait AppDeployment<E: Application, P = LocalClusterProvisioner>: Send + 'static {
    type Handle: AppHandle;

    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

The trait has these properties:

• deploy consumes self. A deployment value describes one preparation attempt and returns its runtime access handle.
• The handle is typed. Handle can be any Clone + Send + Sync + 'static type. Managed lifetime is registered separately; see Handle Ownership and Teardown.
• Clone is required by the factory. with_app needs A: AppDeployment<E> + Clone + Sync because AppDeploymentFactory clones the description on each prepare. Deployment structs should contain configuration such as node counts, ports, and paths rather than live resources.

A minimal implementation, from the kvstore example:

// examples/kvstore/testing/integration/src/app.rs
#[derive(Clone)]
pub struct KvLocalApp {
    deployment: KvTopology,
}

#[async_trait]
impl AppDeployment<AppHostEnv> for KvLocalApp {
    type Handle = LocalAppCluster<KvEnv>;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        ctx.deploy_local_cluster::<KvEnv>(self.deployment).await
    }
}

DeployContext API

One context belongs to one scenario preparation. It carries the active cluster provisioner, outer deployment and clients, exposed handles, and a cleanup stack. Routing every managed child through this context registers cleanup as soon as the resource is acquired, including when deployment fails partway.

deploy does not expose its returned handle. Use it when only the parent needs the child handle. Use deploy_and_expose when workloads should also be able to request the child directly. Both expose and expose_named return AppDeployError::DuplicateHandle if the type or type/name pair is already registered.

Nested Deployments

A deployment composes children by calling ctx.deploy(...) or ctx.deploy_and_expose(...) on other AppDeployment values. The parent decides what is visible:

#[async_trait]
impl AppDeployment<TestEnv> for ParentApp {
    type Handle = ParentHandle;

    async fn deploy(self, ctx: &mut DeployContext<TestEnv>) -> Result<Self::Handle, DynError> {
        let child = ctx.deploy(ChildApp).await?; // child handle NOT exposed
        let parent = ParentHandle { child };

        ctx.expose(parent.clone())?; // only the parent is visible

        Ok(parent)
    }
}

Workloads can then require ParentHandle but not ChildHandle: the child stays an implementation detail. Its managed resources remain registered with scenario cleanup whether or not the returned handle is exposed or embedded. Expose the child too when workloads legitimately need it.

flowchart TD
    Root[Root AppDeployment] -->|deploy| C1[Child A]
    Root -->|deploy_and_expose| C2[Child B]
    Root -->|expose| RH[Root handle]
    C2 --> BH[Child B handle]
    RH --> W[Workloads]
    BH --> W
    RH:::hd
    BH:::hd
    W:::sc
    classDef hd stroke:#4caf7d,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The Outer Scenario: deployment() and node_clients()

For AppHost scenarios, deployment() is the empty AppHostTopology and node_clients() is empty; everything lives in your handles. On a regular uniform-cluster scenario, they are how an app preset wraps the managed cluster itself:

// examples/kvstore/testing/integration/src/app.rs
#[async_trait]
impl AppDeployment<KvEnv> for KvExistingClusterApp {
    type Handle = KvStoreCluster;

    async fn deploy(self, ctx: &mut DeployContext<KvEnv>) -> Result<Self::Handle, DynError> {
        Ok(KvStoreCluster::new(
            ctx.deployment().clone(),
            ctx.node_clients().clone(),
        ))
    }
}

This preset does not launch nodes. It returns typed access to the nodes already managed by the scenario.

Root-Handle Auto-Exposure

After the root deployment returns, AppDeploymentFactory checks ctx.contains::<A::Handle>(). If the root handle type is not already exposed, the factory exposes the returned handle as the default for its type. So:

• A simple root app can just return Ok(handle) and workloads can require_app::<Handle>() with no explicit expose.
• A root app that already exposed its own handle (like the stack apps in Composing Heterogeneous Stacks) is left alone, so there is no duplicate error.

See Also

• AppHost and with_app: how a deployment gets registered and prepared.
• Handle Ownership and Teardown: what exposure means for resource lifetime.
• One Binary: LocalProcessApp, Uniform Child Clusters: LocalAppCluster: ready-made deployments to compose.

Handle Ownership and Teardown

Handles provide typed access to deployed applications. The scenario runtime owns managed resource lifetime separately through a cleanup stack.

Cloning a handle preserves access to its shared state, but does not extend a process or cluster beyond the run that created it.

Typed and Named Handles

The registry keys every handle by concrete type plus name (TypeId and a string). An empty name is the default handle for that type.

Duplicate exposure is an error rather than a replacement. Use distinct names when a scenario exposes multiple values of one handle type, then retrieve them with app_named or require_app_named.

Missing handles are typed runtime errors. require::<T>() and require_named::<T>(name) return AppDeployError::HandleMissing with the requested Rust type and instance name.

What a Handle Means

pub trait AppHandle: Clone + Send + Sync + 'static {}

impl<T> AppHandle for T
where
    T: Clone + Send + Sync + 'static,
{}

An app handle can be a client, a control surface, or a domain aggregate such as JobStackHandle. Retrieval clones it so workloads can use it without borrowing the registry.

The registry lookup uses TypeId. Requesting the wrong concrete type is therefore a runtime miss, not a compile-time error. Prefer specific handle types or domain newtypes over primitives whose role is unclear.

Cloneability is about access, not ownership of the deployment. LocalProcessHandle clones share process state and controls; ClusterHandle clones share clients and control adapters. Scenario cleanup remains authoritative for managed lifetime.

Managed Lifetime

Every framework adapter that acquires a managed resource registers a cleanup guard immediately. DeployContext collects those guards in acquisition order and transfers the stack to the scenario runtime after successful preparation.

flowchart LR
    D["deploy child"] --> G["register cleanup guard"]
    G --> H["return and optionally expose handle"]
    H --> R["scenario runs"]
    R --> C["cleanup stack: last acquired, first released"]

This produces two parallel structures:

The cleanup stack decides when managed resources stop. A handle clone retained outside the registry does not postpone cleanup; after cleanup, operations on a LocalProcessHandle fail because the run no longer owns the process.

Dependency-Ordered Teardown

Deploy dependencies before dependents:

let queue = ctx.deploy_and_expose(QueueLocalApp::nodes(2)).await?;
let results = ctx.deploy_and_expose(KvLocalApp::nodes(2)).await?;
let worker = ctx
    .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))
    .await?;

Each deployment registers cleanup as soon as it acquires its resource. LIFO cleanup therefore stops the worker first, then the result store, then the queue. The order is independent of which handles the final JobStackHandle embeds or how many clones workloads retain.

Exposure order usually follows acquisition order, but it is not the ownership mechanism. Expose a handle when test code needs to find it; register cleanup when the framework acquires a managed resource.

Partial-Deployment Failure

If deployment fails halfway through, dropping DeployContext runs every cleanup guard already registered. The same LIFO rule applies, so successfully started dependents stop before their dependencies even though no scenario runner was created.

Readiness belongs inside deployment for the same reason. LocalProcessApp::with_readiness stops its just-started process if the check fails, while the context cleans up all earlier children.

Custom AppDeployment implementations should acquire managed resources through framework adapters such as LocalProcessApp and deploy_cluster. A raw process started directly by application code has no cleanup guard unless that code implements and registers an adapter.

Manual Control During a Run

Automatic teardown does not prevent explicit control. A workload can call stop, start, or restart on a process handle, or the corresponding node methods on a cluster handle. Cleanup remains registered and idempotently closes whatever is still active when the run ends.

Managed applications therefore support both properties:

• test code can deliberately change runtime state;
• every exit path still has a final owner that cleans up.

Keeping Artifacts

Managed cleanup normally removes generated working directories. Use LocalProcessApp::keep_tempdir(true) or LocalProcessHandle::keep_tempdir() for a process. Primary-cluster artifact retention is controlled by the deployment policy described in Readiness, Retry, and Cleanup.

See Also

• AppDeployment and DeployContext: where children, handles, and cleanup are assembled.
• One Binary: LocalProcessApp: a managed process and its control handle.
• Shared Cluster Provisioning: cluster handles across ownership modes.

One Binary: LocalProcessApp

LocalProcessApp<C> deploys one local binary with a typed client, without modeling it as a node topology.

Application code supplies the launch files, client type, and readiness check. The framework manages the process lifetime, working directory, and teardown. This is used for third-party infrastructure such as a message broker or database, and for singleton services such as a sequencer or indexer inside a composed stack.

Construction

LocalProcessApp::new(label, launch, endpoints, client)

LaunchSpec (from testing_framework_runner_local) is a plain launch plan:

NodeEndpoints describes where the process listens: an api: SocketAddr plus extra_ports keyed by NodeEndpointPort (TestingApi, Network, or Custom(String)). Build one with NodeEndpoints::from_api_port(port) and insert_port.

Endpoints are declared, not allocated. The generic process layer does not select ports. The launch configuration and the supplied endpoints must use the same values.

Builder Options

If the readiness closure fails, deployment returns an error, stops the new process, and cleans up children deployed earlier (see Handle Ownership and Teardown).

Example: A Single nats-server Process

The nats example normally runs as a uniform cluster, but its NatsClient works just as well against one broker started as a process app:

use std::time::Duration;

use nats_runtime_ext::NatsClient;
use testing_framework_app::LocalProcessApp;
use testing_framework_runner_local::{LaunchSpec, NodeEndpoints};

let launch = LaunchSpec {
    binary: std::env::var("NATS_SERVER_BIN")?.into(),
    args: vec!["-p".into(), "4222".into(), "-m".into(), "8222".into()],
    ..LaunchSpec::default()
};

let client = NatsClient::new(
    "nats://127.0.0.1:4222".to_owned(),
    "http://127.0.0.1:8222".parse()?,
);

let broker = LocalProcessApp::new("nats", launch, NodeEndpoints::from_api_port(8222), client)
    .with_readiness(|_endpoints, client| async move {
        for _ in 0..50 {
            if client.is_healthy().await.unwrap_or(false) {
                return Ok(());
            }
            tokio::time::sleep(Duration::from_millis(100)).await;
        }
        Err("nats-server did not become healthy".into())
    });

broker is an AppDeployment for any environment, so a root deployment composes it like any other child:

let nats = ctx.deploy_and_expose(broker).await?;

The Handle: LocalProcessHandle

deploy returns LocalProcessHandle<C>. Clones share access to the same process state, while scenario cleanup owns the process lifetime. Cleanup stops the process even if a handle clone still exists.

A workload retrieves the handle like any other (see AppHost and with_app):

let broker = ctx.require_app::<LocalProcessHandle<NatsClient>>()?;
broker.restart().await?;
assert!(broker.is_running().await);

Tests can use start, restart, and stop for process lifecycle and fault injection. These operations fail after scenario cleanup has closed the managed resource.

See Also

• AppDeployment and DeployContext: composing a process app under a root deployment.
• Composing Heterogeneous Stacks: mixing single processes with child clusters.

Uniform Child Clusters: LocalAppCluster

LocalAppCluster<E> runs an additional uniform cluster of local processes as one child of a composed stack.

For N identical nodes of one binary with peer wiring and per-node clients, use ScenarioBuilder<E> when the cluster is the system under test. When the cluster is one component of a larger stack, deploy it as a LocalAppCluster inside the root deployment.

The environment E must implement LocalDeployerEnv (config rendering, ports, process spec; see Local Deployer). That work is the same whether the app runs standalone or as a child, so a cluster env written for uniform scenarios is reusable here unchanged.

Starting a Child Cluster

Inside an AppDeployment, DeployContext::deploy_local_cluster launches every node described by the deployment (node-0, node-1, …), waits for network readiness, registers cleanup, and returns the cluster handle:

// examples/kvstore/testing/integration/src/app.rs
#[derive(Clone)]
pub struct KvLocalApp {
    deployment: KvTopology,
}

impl KvLocalApp {
    #[must_use]
    pub fn nodes(nodes: usize) -> Self {
        Self { deployment: KvTopology::new(nodes) }
    }
}

#[async_trait]
impl AppDeployment<AppHostEnv> for KvLocalApp {
    type Handle = LocalAppCluster<KvEnv>;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        ctx.deploy_local_cluster::<KvEnv>(self.deployment).await
    }
}

The kvstore preset delegates cluster provisioning to deploy_local_cluster, which registers a cleanup guard and returns a cloneable access and control handle. Scenario cleanup stops any remaining nodes independently of handle clones.

The Handle API

Node names follow the node-{index} convention used at startup. LocalAppCluster<E> is the backend-independent ClusterHandle<E> alias; it exposes the supported common control surface rather than an underlying ManualCluster.

Per-node control is provided by the cluster handle. A workload restarting a child-cluster node does not need the scenario-level with_node_control() capability.

Worked Example: kvstore Convergence Across a Restart

The kvstore_app_host_convergence bin runs a three-node kv cluster as an app, then exercises convergence across a node restart:

// examples/kvstore/examples/src/bin/app_host_convergence.rs
let mut scenario = AppHost::scenario()
    .with_app(KvLocalApp::nodes(3))
    .with_run_duration(Duration::from_secs(5))
    .with_workload(KvAppHostConvergence::new(3))
    .build()?;

let deployer = AppHostLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

The workload requires the cluster handle and drives it directly:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<LocalAppCluster<KvEnv>>()?;

    ensure_cluster_shape(&cluster, self.expected_nodes)?;
    put_value(&cluster, "before-restart").await?;
    cluster.restart_node("node-0").await?;
    cluster.wait_node_ready("node-0").await?;
    put_value(&cluster, "after-restart").await?;

    Ok(())
}

put_value writes through cluster.first_client(); ensure_cluster_shape checks node_count(), clients(), node_client("node-0"), and node_pid("node-0"). Run it with:

cargo run -p kvstore-examples --bin kvstore_app_host_convergence

The kvstore environment resolves its node binary through a fallback provider chain, so this example does not require a manually configured binary path (see Binary Providers).

Exposing the Cluster to Workloads

KvLocalApp returns the raw LocalAppCluster<KvEnv> as its handle, and the factory auto-exposes it, so workloads request LocalAppCluster<KvEnv> directly. In a composed stack you can either expose the raw cluster handle (as the job stack does), wrap it in a domain newtype (StoreHandle) for clearer requirements, or use named handles when two child clusters share an environment type (see Composing Heterogeneous Stacks).

See Also

• One Binary: LocalProcessApp: the single-process counterpart.
• Backend Scope: why child clusters are local-only today.

Composing Heterogeneous Stacks

A root AppDeployment deploys the components, passes dependency addresses between them, and exposes typed handles.

A scenario accepts one with_app registration (see AppHost and with_app), so the root deployment composes its children. It exposes component handles needed by workloads and may also expose an aggregate stack handle. The examples/multi_app fixture contains a queue cluster, job-worker process, and kv result-store cluster in one job-processing pipeline.

The Root App

// examples/multi_app/fixture/src/lib.rs
#[derive(Clone)]
struct JobStackApp {
    queue_nodes: usize,
    result_nodes: usize,
}

impl JobStackApp {
    fn new() -> Self {
        Self {
            queue_nodes: 2,
            result_nodes: 2,
        }
    }
}

#[async_trait]
impl AppDeployment<AppHostEnv> for JobStackApp {
    type Handle = JobStackHandle;

    async fn deploy(self, ctx: &mut DeployContext<AppHostEnv>) -> Result<Self::Handle, DynError> {
        let queue = ctx
            .deploy_and_expose(QueueLocalApp::nodes(self.queue_nodes))
            .await?;
        let results = ctx
            .deploy_and_expose(KvLocalApp::nodes(self.result_nodes))
            .await?;

        let queue_url = queue
            .first_client()
            .ok_or("queue cluster has no clients")?
            .base_url()
            .clone();
        let results_url = results
            .first_client()
            .ok_or("result store has no clients")?
            .base_url()
            .clone();
        let worker = ctx
            .deploy_and_expose(JobWorkerApp::new(queue_url, results_url))
            .await?;

        let stack = JobStackHandle { queue, results, worker };
        ctx.expose(stack.clone())?;

        Ok(stack)
    }
}

The example establishes these relationships:

• Children are deployed through the context (deploy_and_expose), so each cluster and the worker process are owned by the runtime for the whole run.
• Dependencies are constructor arguments. The worker receives the queue and result-store URLs from the already-deployed clusters, so the root deployment shows the dependency graph.
• Both levels are exposed: each component handle and the aggregate JobStackHandle, allowing workloads to request the smallest handle they need.

flowchart TD
    Root[JobStackApp] --> Q["queue cluster x2<br/>LocalAppCluster&lt;QueueEnv&gt;"]
    Root --> R["result store x2<br/>LocalAppCluster&lt;KvEnv&gt;"]
    Q --> W["job worker<br/>LocalProcessApp"]
    R --> W
    Q --> St[JobStackHandle]
    R --> St
    W --> St
    Q:::cl
    R:::cl
    W:::pr
    St:::hd
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef pr stroke:#e08a3c,stroke-width:2.5px;
    classDef hd stroke:#4caf7d,stroke-width:2.5px;

Workloads Require What They Need

Each workload asks for exactly the handles it uses, the whole stack or one component:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let stack = ctx.require_app::<JobStackHandle>()?;
    let queue = stack.queue.first_client().ok_or("queue cluster has no clients")?;

    for index in 0..self.count {
        let response: EnqueueResponse = queue
            .post("/queue/enqueue", &EnqueueRequest { payload: job_key(index) })
            .await?;
        if !response.accepted {
            return Err(format!("queue rejected job {index}").into());
        }
    }
    Ok(())
}

Assembling and running the scenario is unchanged from any other AppHost run:

let mut scenario = AppHost::scenario()
    .with_app(JobStackApp::new())
    .with_run_duration(Duration::from_secs(10))
    .with_workload(EnqueueJobs::new(10))
    .with_expectation(AllJobsCompleted::new(10))
    .build()?;

let deployer = AppHostLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

cargo test -p multi-app-e2e

Wiring Dependencies Between Components

Pass dependencies through constructors. Deploy the dependency first and pass its handle or address into the dependent’s constructor, as JobWorkerApp::new(queue_url, results_url) does above. This records the dependency graph and acquisition order in the root deployment. A child can call ctx.require::<T>(), but then it depends on another deployment having exposed T earlier. If that did not happen, deployment fails at run time with HandleMissing.

The same rule applies to process-level wiring. The job worker is a LocalProcessApp whose LaunchSpec receives the queue and store URLs as command-line arguments: deploy the dependency, read its client’s base_url(), and feed the address into the process. Do not have the process guess.

Use named handles for two instances of one type. The registry allows one default handle per concrete type; a second expose of the same type is a duplicate error (see Handle Ownership and Teardown). Two kv clusters in one stack therefore need names:

ctx.expose_named("primary", primary)?;
ctx.expose_named("replica", replica)?;

// in the workload:
let primary = ctx.require_app_named::<LocalAppCluster<KvEnv>>("primary")?;

Expose components as well as the stack when both are used. A workload that touches one component can request its handle directly, while stack-level workloads can request the aggregate handle.

See Also

• AppDeployment and DeployContext: the composition API in detail.
• Uniform Child Clusters: LocalAppCluster: the child clusters used here.
• Backend Scope: where composed stacks can run today.

Backend Scope

The app layer currently deploys components only through the local backend. Compose and Kubernetes support uniform single-application scenarios.

This chapter lists the supported combinations and the APIs missing from the container backends.

What Works Where

An app preset that deploys nothing, such as the “existing cluster” presets in AppHost and with_app, works on every backend because it only wraps ctx.deployment() and ctx.node_clients() in a typed handle. Deploying new components through the app layer is local-only: LocalProcessApp and LocalAppCluster use the local deployer’s process primitives (ProcessNode, ManualCluster, ProcessDeployer), and AppHostLocalDeployer is a local process deployer.

Single-app Compose and Kubernetes deployers are unchanged by the app layer. The kvstore and OpenRaft examples keep dedicated bins for them (kvstore_compose_convergence, kvstore_k8s_convergence, openraft_kv_compose_failover, openraft_kv_k8s_failover); see Compose Deployer and Kubernetes Deployer.

Why the Gap Exists

The app layer starts, addresses, and stops individual units and can run application code between those starts, for example to check readiness or pass an address to a dependent component. The local deployer provides per-unit APIs. Compose and Kubernetes currently render and deploy a complete uniform scenario as one planned unit. Supporting AppDeployment on those backends requires corresponding per-unit planning and deployment APIs.

Choosing a Shape Today

flowchart TD
    Q{System under test} -->|one uniform cluster| U[ScenarioBuilder over a topology]
    Q -->|composed stack| A[AppHost + root AppDeployment]
    U --> B{Backend}
    B --> L1[Local]
    B --> C1[Compose]
    B --> K1[Kubernetes]
    A --> L2[Local only]

• Composed stacks: run locally. The local backend provides direct process control and per-unit restarts for heterogeneous stacks (see Composing Heterogeneous Stacks).
• Uniform single-app scenarios: use any supported backend. A topology containing one application can run locally, with Compose, or on Kubernetes, subject to the capability matrix.
• Split suites by shape, not by app. If one system needs both a composed integration stack and a large containerized soak test of its main cluster, express them as two scenarios: an AppHost stack running locally, and a uniform scenario of the main app running on Compose/K8s. The kvstore example uses one environment crate with separate binaries per shape and backend.

Keep Workloads Backend-Independent

Write workloads against typed handles and clients, not against backend details. A workload that requires a StoreHandle does not care whether the store came from a LocalAppCluster today or a future containerized unit:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let store = ctx.require_app::<StoreHandle>()?; // no backend visible here
    store.put("/kv/scope-check", "ok").await?;
    Ok(())
}

If another backend later supports app composition, backend-specific changes should remain in the root deployment and its child adapters. Workloads and expectations can continue using the same handles. The OpenRaft “existing cluster” preset already works with Compose and Kubernetes because it only reads ctx.deployment() and ctx.node_clients().

Note: node-control-style fault injection inside a composed stack (restart one child-cluster node) is a handle method on LocalAppCluster, so it is local-only by construction. Fault injection on containerized uniform scenarios goes through the scenario-level node control capability instead; see the openraft openraft_kv_k8s_failover bin.

See Also

• Capability Matrix: the full feature-by-backend table.
• Local Deployer: the backend the app layer builds on.
• AppHost and with_app: the entry point this scope applies to.

Part III — Scenario Runtime

These chapters describe what happens while a scenario runs: traffic, verification, capabilities, and observation.

The same runtime serves all three declarative entry patterns: uniform clusters, composed app stacks, and attached external nodes.

• Workloads and Concurrency — driving the system under test
• Expectations and Evaluation — verifying outcomes
• The Verb Layer — concise domain actions over the explicit builder API
• Scenario Capabilities — capability-gated features such as node control
• Chaos and Controlled Failure — restarts and failover from workloads
• Runtime Extensions — typed scenario-lifetime services
• Continuous Observation — snapshots, history, and event streams for test logic
• Telemetry and External Observability — metrics, logs, and tracing

Workloads and Concurrency

Workloads describe the activity a scenario generates: every workload runs as its own concurrent task against the shared RunContext, and the runner decides when the run window ends.

The Workload Trait

A workload is any type implementing Workload<E> from testing-framework-core (testing-framework/core/src/scenario/workload.rs):

use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext, Workload};

#[async_trait]
pub trait Workload<E: Application>: Send + Sync {
    fn name(&self) -> &str;

    fn expectations(&self) -> Vec<Box<dyn Expectation<E>>> {
        Vec::new()
    }

    fn init(
        &mut self,
        _descriptors: &E::Deployment,
        _run_metrics: &RunMetrics,
    ) -> Result<(), DynError> {
        Ok(())
    }

    async fn start(&self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

The trait methods are:

• name identifies the workload in logs and failure reports.
• expectations lets a workload attach its own checks. with_workload collects them into the scenario alongside explicitly added expectations (see Expectations and Evaluation).
• init runs synchronously at build() time, before anything is deployed. It receives the resolved deployment descriptors and the RunMetrics (run duration). A failing init aborts the build with a WorkloadInit error.
• start is the async body. It runs once per scenario run and must return when its work is done.

The runner schedules every workload and applies the same concurrency, panic capture, and run-window behavior described below.

Register workloads on any builder with .with_workload(w) or .with_workload_boxed(boxed).

How the Runner Executes Workloads

The runner (testing-framework/core/src/scenario/runtime/runner.rs) drives a run in fixed phases:

flowchart LR
    P[start_capture<br/>expectations]:::sc --> W[Workload window<br/>run_duration]:::sc
    W --> C[Cooldown window]:::sc
    C --> D[Drain remaining<br/>workloads]:::sc
    D --> S[Settle wait]:::sc
    S --> E[Evaluate<br/>expectations]:::sc

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The runner uses the following concurrency rules:

• All workloads run concurrently. Each workload is spawned into its own Tokio task via a JoinSet; there is no ordering between them.
• Panics become errors. A panicking workload does not abort the process; the panic is caught and reported as workload panicked: <message>.
• One failure fails the run. The runner joins workload tasks as they finish. The first workload that returns Err (or panics) ends the run immediately with ScenarioError::Workload; expectations are not evaluated.
• Finishing early ends the window early. If every workload returns Ok before with_run_duration elapses, the workload phase completes without waiting out the timer.
• The run duration sets the maximum workload window but does not cancel workloads. When every workload finishes early, the window ends early and cooldown begins. When the timer expires while workloads are still running, the runner keeps driving them through the cooldown window and then waits for them to finish (drain_workloads). A workload that never returns blocks the run indefinitely.

Treat with_run_duration as the guaranteed run window, not as a workload timeout. A long-running workload should bound its own work, either by operation count or by reading ctx.run_duration() and stopping at the deadline.

After the workload window, managed deployments get a cooldown window (minimum 30 seconds when the framework owns the node lifecycle) plus a short settle wait so runtime extensions catch up before evaluation. Both are tuned with with_expectation_cooldown; see Expectations and Evaluation.

Worked Example: a Key/Value Write Workload

The kvstore example ships KvWriteWorkload (examples/kvstore/testing/workloads/src/write.rs), a rate-limited writer over the node HTTP clients:

use async_trait::async_trait;
use kvstore_runtime_ext::KvEnv;
use testing_framework_core::scenario::{DynError, RunContext, Workload};

#[async_trait]
impl Workload<KvEnv> for KvWriteWorkload {
    fn name(&self) -> &str {
        "kv_write_workload"
    }

    async fn start(&self, ctx: &RunContext<KvEnv>) -> Result<(), DynError> {
        let clients = ctx.node_clients().snapshot();
        let Some(leader) = clients.first() else {
            return Err("no kv node clients available".into());
        };

        for idx in 0..self.operations {
            let key = format!("{}-{}", self.key_prefix, idx % self.key_count);
            let response: PutResponse = leader
                .put(&format!("/kv/{key}"), &PutRequest { value: format!("value-{idx}"), expected_version: None })
                .await?;

            if !response.applied {
                return Err(format!("leader rejected write for key {key}").into());
            }

            if let Some(delay) = interval {
                tokio::time::sleep(delay).await;
            }
        }

        Ok(())
    }
}

This workload takes one client snapshot, runs a bounded number of operations, controls its rate with a sleep, and returns Err for an unexpected response so the runner stops the run.

The workload is bounded by self.operations, so it terminates on its own; the run duration only decides how long the scenario stays up around it.

Accessing the RunContext

RunContext<E> (testing-framework/core/src/scenario/runtime/context.rs) gives a workload access to:

Notes on the client surface:

• node_clients().snapshot() clones the current client vector so you can iterate across .await points. Use with_clients(|clients| ...) for synchronous reads without the clone.
• extension::<T>() returns a clone of a value registered by a runtime extension factory, for example an ObservationHandle from Continuous Observation.
• node_control() is only populated when the scenario was built with the node-control capability; see Scenario Capabilities and Chaos and Controlled Failure.

Workloads in app-layer scenarios additionally use AppRunContextExt (from testing-framework-app) to reach composed application handles:

use testing_framework_app::AppRunContextExt;

let cluster = ctx.require_app::<KvStoreCluster>()?;

OpenRaftKvClusterAccessible (examples/openraft_kv/testing/workloads/src/handle_access.rs) uses only require_app to assert that the exposed cluster handle matches the expected topology. See AppHost and with_app for the app layer itself.

See Also

• Expectations and Evaluation — the checks that run after your traffic
• Runtime Extensions — sharing typed values with workloads
• Chaos and Controlled Failure — workloads that restart nodes
• Continuous Observation — polling application state while workloads run

Expectations and Evaluation

Expectations define success conditions. They can capture state before workloads start, check invariants while traffic runs, and evaluate the final state after the run settles.

The Expectation Trait

Expectation<E> lives in testing-framework/core/src/scenario/expectation.rs:

use async_trait::async_trait;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};

#[async_trait]
pub trait Expectation<E: Application>: Send + Sync {
    fn name(&self) -> &str;

    fn init(
        &mut self,
        _descriptors: &E::Deployment,
        _run_metrics: &RunMetrics,
    ) -> Result<(), DynError> {
        Ok(())
    }

    async fn start_capture(&mut self, _ctx: &RunContext<E>) -> Result<(), DynError> {
        Ok(())
    }

    /// Optional periodic check used by fail-fast expectation mode.
    async fn check_during_capture(&mut self, _ctx: &RunContext<E>) -> Result<(), DynError> {
        Ok(())
    }

    async fn evaluate(&mut self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

The trait methods are:

• init runs at build() time with the resolved deployment and run metrics; a failure aborts the build.
• start_capture runs once per expectation before any workload starts. Use it to record a baseline (initial counters, starting state). A failure here is ScenarioError::ExpectationCapture and stops the run before traffic begins.
• check_during_capture is a fail-fast hook. The runner calls it on every expectation roughly once per second for the whole workload window (and the cooldown window). The default is a no-op, so existing end-of-run expectations are unaffected. The first check that returns Err aborts the run immediately with ScenarioError::ExpectationFailedDuringCapture. Use it for invariants that must hold throughout the run.
• evaluate checks the final condition after the run settles. It takes &mut self, so it can consume state accumulated during capture.

Registration

Two paths feed the scenario’s expectation list:

1. Explicit: .with_expectation(exp) or .with_expectation_boxed(boxed) on any builder.
2. Workload-attached: when you call .with_workload(w), the builder also collects w.expectations() (see Workloads and Concurrency). The default implementation returns none.

Both end up in the same list and are treated identically at run time.

Workload-attached expectations let a workload register the checks associated with its own traffic. Adding the workload also adds those checks.

Evaluation and Failure Aggregation

flowchart LR
    SC[start_capture]:::sc --> W[Workload window<br/>+ periodic checks]:::sc
    W --> CD[Cooldown + settle]:::sc
    CD --> EV[evaluate all]:::sc
    EV --> R{failures?}
    R -->|no| OK[run passes]
    R -->|yes| AGG[aggregated report]

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

At the end of the run the runner evaluates every registered expectation, even after failures. Each failure is recorded as name: error, and the results are joined into a single ScenarioError::Expectations report:

expectations failed:
kv_converges: kv convergence not reached within 20s for 20 keys
openraft_kv_converges: timed out waiting for observed replicated state convergence ...

This is different from workload failures and capture-check failures, which abort immediately.

Cooldown: with_expectation_cooldown

Workload traffic may need time to settle before evaluation: replication lags, queues drain, and restarted nodes rejoin. The builder exposes:

.with_expectation_cooldown(Duration::from_secs(20))

Verified behavior (runner.rs and definition/validation.rs):

• If you never call it, the cooldown defaults to 10 seconds. (build() also enforces a minimum run duration of 10 seconds.)
• After the workload window, the runner keeps the run alive for the cooldown window, still joining unfinished workloads and still running check_during_capture ticks.
• When the framework owns the node lifecycle (managed clusters), the cooldown window is raised to a minimum of 30 seconds so restarted or freshly deployed nodes stabilize.
• Before calling evaluate, the runner additionally sleeps a short settle wait derived from the same setting (at least 2 seconds when a cooldown is configured or node control is active) so runtime extensions such as observers catch up.

Set the cooldown to zero only for scenarios without managed nodes where staleness cannot matter.

Worked Example: Convergence Checks

The kvstore example’s KvConverges (examples/kvstore/testing/workloads/src/expectations.rs) is a plain polling expectation. It only implements evaluate and does its own retry loop against the node clients:

use async_trait::async_trait;
use kvstore_runtime_ext::KvEnv;
use testing_framework_core::scenario::{DynError, Expectation, RunContext};

#[async_trait]
impl Expectation<KvEnv> for KvConverges {
    fn name(&self) -> &str {
        "kv_converges"
    }

    async fn evaluate(&mut self, ctx: &RunContext<KvEnv>) -> Result<(), DynError> {
        let clients = ctx.node_clients().snapshot();
        if clients.is_empty() {
            return Err("no kv node clients available".into());
        }

        let deadline = tokio::time::Instant::now() + self.timeout;
        while tokio::time::Instant::now() < deadline {
            if self.is_converged(&clients).await? {
                return Ok(());
            }
            tokio::time::sleep(self.poll_interval).await;
        }

        Err(format!(
            "kv convergence not reached within {:?} for {} keys",
            self.timeout, self.key_count
        )
        .into())
    }
}

The example follows two conventions:

• Poll with a deadline inside evaluate. Eventual consistency is the common case; a one-shot read makes flaky tests.
• Make the error message carry the diagnosis. State what was expected, how long you waited, and (where available) what was last observed.

The openraft_kv variant, OpenRaftKvConverges (examples/openraft_kv/testing/workloads/src/convergence.rs), reads the cluster observer registered as a runtime extension instead of querying nodes directly:

async fn evaluate(&mut self, ctx: &RunContext<OpenRaftKvEnv>) -> Result<(), DynError> {
    let expected = expected_kv(&self.key_prefix, self.total_writes);
    let observer = ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>()?;

    wait_for_observed_replication(&observer, &expected, self.timeout).await?;

    Ok(())
}

The observer polls every node in the background, and the expectation waits for a matching snapshot without maintaining its own client polling state. See Continuous Observation for the mechanism.

See Also

• Workloads and Concurrency — the traffic these checks judge
• Continuous Observation — snapshot-based state for expectations
• Runtime Extensions — how extension handles reach evaluate
• Telemetry and External Observability — asserting on Prometheus metrics

The Verb Layer

The verb layer provides optional, domain-specific helpers for recurring test actions. It uses the same scenario builder, workloads, expectations, and capabilities described in the preceding chapters.

Two Equivalent Levels

The explicit API names the objects being assembled:

let scenario = QueueScenarioBuilder::with_deployment(QueueTopology::new(5))
    .with_node_control()
    .with_network_control()
    .with_workload(QueueProduceWorkload::new().operations(400).rate_per_sec(40))
    .with_workload(RandomRestartWorkload::new(
        Duration::from_secs(5),
        Duration::from_secs(15),
        Duration::from_secs(15),
    ))
    .with_workload(NetworkPartitionWorkload::new(
        NetworkPartitionSpec::new([
            vec!["node-0", "node-1"],
            vec!["node-2", "node-3", "node-4"],
        ]),
        Duration::from_secs(20),
        Duration::from_secs(60),
    ))
    .with_expectation(QueueConverges::new(400).timeout(Duration::from_secs(60)))
    .with_run_duration(Duration::from_secs(120))
    .build()?;

The verb API lowers to those same operations:

QueueScenario::nodes(5)
    .produce(400).rate_per_sec(40).done()
    .restart_nodes_randomly().every_secs(5, 15).done()
    .partition(["node-0", "node-1"], ["node-2", "node-3", "node-4"])
        .hold_secs(20).done()
    .expect_converged(400).within_secs(60)
    .run_secs(120)
    .await?;

The explicit API remains available at every point. Use it for one-off workloads, unusual policies, or operations that do not have a domain verb.

How Verbs Map to the Builder

A verb does not introduce a second runtime. Its sub-builder stores an ordinary workload or expectation and adds it when done() or a terminal method is called:

pub trait QueueDslExt: CoreBuilderAccess<Env = QueueEnv> + Sized {
    fn produce(self, operations: usize) -> QueueProduceBuilder<Self> {
        QueueProduceBuilder {
            builder: self,
            workload: QueueProduceWorkload::new().operations(operations),
        }
    }
}

impl<B: CoreBuilderAccess<Env = QueueEnv>> QueueProduceBuilder<B> {
    pub fn done(self) -> B {
        self.builder.map_core_builder(|builder| {
            builder.with_workload(self.workload)
        })
    }
}

Both forms therefore use the same execution, failure aggregation, and teardown. Generic and application-specific verbs can extend the same builder chain.

Capability-Aware Verbs

Some actions require a runtime capability. The verb should request it when the requirement follows directly from the action:

• restart_nodes_randomly() transitions a plain builder to a node-control builder.
• partition(...).done() requests network control before adding the partition workload.
• A data-plane verb such as produce(...) needs no control capability.

The resulting Rust type records the capability transition. If a deployer cannot supply the requested capability, deployment fails before the workload starts.

Do not hide an unrelated policy choice inside a verb. A verb may request what its action necessarily needs; retry policy, cleanup policy, backend selection, and other test-wide decisions remain explicit.

Designing Application Verbs

Put vocabulary shared by applications in the framework and vocabulary specific to one protocol in that application’s testing crate. A verb should:

1. names an operation in the application’s domain;
2. configures one workload or expectation, or a small fixed combination;
3. exposes meaningful options through a short sub-builder;
4. returns the underlying builder through done() or a clear terminal method;
5. preserves access to with_workload and with_expectation for uncommon cases.

The queue example keeps partition and random restarts generic, while produce and expect_converged live with the queue integration. Other applications can then reuse chaos behavior without depending on queue terminology.

See Also

• Workloads and Concurrency and Expectations and Evaluation: the objects verbs add.
• Scenario Capabilities: the requirements capability-aware verbs request.
• Chaos and Controlled Failure: the generic restart and partition workloads.

Scenario Capabilities

Capabilities record, in the type system, which deployer services a scenario requests, such as node control or external telemetry. Unsupported combinations fail during construction, compilation, or deployment rather than during a workload.

The Capability Type Parameter

The core builder is generic over a capability marker: Builder<E, Caps> with Caps = () by default. Building a scenario produces Scenario<E, Caps>, and deployers are typed as Deployer<E, Caps>, so a deployer that cannot provide a capability does not accept scenarios that demand it.

The public wrappers (testing-framework/core/src/scenario/definition/builder.rs):

All three expose the same fluent surface (with_workload, with_expectation, with_run_duration, …), so the capability switch can happen anywhere in the chain:

let scenario = ScenarioBuilder::with_deployment(topology)
    .with_node_control()                 // () -> NodeControlCapability
    .with_workload(my_restart_workload)
    .with_run_duration(Duration::from_secs(60))
    .build()?;

RequiresNodeControl (testing-framework/core/src/scenario/capabilities.rs) is how build() and deployers reason about the marker:

pub trait RequiresNodeControl {
    const REQUIRED: bool;
}
// (): false    NodeControlCapability: true    ObservabilityCapability: false

build() uses it to validate the source configuration: a scenario that requires node control but only has external, uncontrolled nodes fails with a SourceConfiguration error (“node control is not available for cluster mode ‘external-only’ …”). See Existing and External Clusters.

Node Control Without ManualCluster

Restarting nodes from a declarative workload does not require ManualCluster. The node-control capability provides access instead:

1. Call .with_node_control() on the builder.
2. Deploy with a deployer that supports the capability (local ships full node control, compose supports restart; the k8s deployer wires no node control handle into managed scenarios, so use its ManualCluster mode instead).
3. Inside a workload, take the handle from the context:

let Some(control) = ctx.node_control() else {
    return Err("this workload requires node control".into());
};

control.restart_node("node-1").await?;

ManualCluster is the imperative API for tests that control the entire node lifecycle themselves; see ManualCluster: Imperative Node Control. The scenario form above runs workloads, expectations, and teardown through the scenario runtime. Chaos and Controlled Failure shows a full failover scenario built this way.

NodeControlHandle

NodeControlHandle (testing-framework/core/src/scenario/control.rs) is the deployer-agnostic control surface. Every method has a default implementation returning a “not supported by this deployer” error, so partial support is explicit at run time:

StartedNode<E> is a plain pair: the node name and a fresh E::NodeClient.

ClusterWaitHandle<E> is the matching wait surface: a single wait_network_ready() used for readiness gates. It is exposed publicly on the runner as Runner::wait_network_ready() (before run starts) and on ManualCluster; inside workloads, prefer waiting on observed application state instead.

StartNodeOptions

StartNodeOptions<E> customizes a dynamic start or restart. Overview (full treatment in Part IV: Ports, Peers, Node Config, and Readiness and Persistence, Snapshots, and Recovery Testing):

The Observability Capability

ObservabilityCapability carries optional telemetry endpoints (Prometheus query URL, OTLP ingest URL, Grafana URL). It does not require node control and is populated through ObservabilityBuilderExt (testing-framework/core/src/scenario/builder_ext.rs):

use testing_framework_core::scenario::ObservabilityBuilderExt;

let builder = ScenarioBuilder::with_deployment(topology)
    .with_metrics_query_url_str("http://127.0.0.1:9090");

Each method transitions ScenarioBuilder<E> into ObservabilityScenarioBuilder<E> (and is a plain setter if you are already there). Url-typed, _str (panicking), and try_..._str (fallible) variants exist for all three endpoints. Deployers merge these values with environment variables; the details, including what telemetry is and is not, are in Telemetry and External Observability.

Capabilities use one marker per scenario, not a set. Choosing with_node_control() gives the scenario node control; choosing an observability method supplies telemetry endpoints. Each deployer declares which Caps it supports; the Capability Matrix lists the available combinations.

See Also

• Chaos and Controlled Failure — node control from workloads
• ManualCluster: Imperative Node Control — the imperative alternative
• Telemetry and External Observability — the observability capability in use
• Capability Matrix — deployer support by capability

Chaos and Controlled Failure

Chaos scenarios deliberately stop or restart nodes and then check recovery. Ordinary workloads perform these operations through the node-control capability.

The Shape of a Chaos Scenario

A chaos test is three ordinary pieces wired together:

1. A scenario built with .with_node_control() (see Scenario Capabilities).
2. A workload that drives traffic, disrupts a node via ctx.node_control(), waits for recovery, and drives traffic again.
3. An expectation that verifies the end state converged despite the disruption.

flowchart LR
    T[Drive traffic]:::sc --> D[Disrupt<br/>restart node]:::sc
    D --> W[Wait for recovery<br/>observed state]:::sc
    W --> T2[Drive traffic again]:::sc
    T2 --> V[Expectation:<br/>state converged]:::sc

    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

No ManualCluster is involved: the deployer provides a NodeControlHandle because the scenario declared the capability.

Worked Example: OpenRaft Leader Failover

The openraft_kv failover test bootstraps a three-node Raft cluster, expands it to three voters, writes a batch, restarts the leader, then writes a second batch through the node elected next. Its workload is in examples/openraft_kv/testing/workloads/src/failover.rs:

#[async_trait]
impl Workload<OpenRaftKvEnv> for OpenRaftKvFailoverWorkload {
    fn name(&self) -> &str {
        "openraft_kv_failover_workload"
    }

    async fn start(&self, ctx: &RunContext<OpenRaftKvEnv>) -> Result<(), DynError> {
        let clients = ctx.node_clients().snapshot();
        let observer = ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>()?;

        ensure_cluster_size(&clients, 3)?;
        self.bootstrap_cluster(&clients).await?;

        let initial_leader = wait_for_observed_leader(&observer, self.timeout, None).await?;
        let membership = OpenRaftMembership::discover(&clients).await?;

        self.promote_cluster(&observer, &clients, initial_leader, &membership).await?;
        self.write_initial_batch(&clients, initial_leader).await?;

        let new_leader = self
            .restart_leader_and_wait_for_failover(ctx, &observer, initial_leader)
            .await?;
        self.write_second_batch(&clients, new_leader).await?;

        Ok(())
    }
}

The disruption itself is a few lines:

let Some(control) = ctx.node_control() else {
    return Err("openraft failover workload requires node control".into());
};

control.restart_node(&format!("node-{leader_id}")).await?;

let new_leader = wait_for_observed_leader(observer, self.timeout, Some(leader_id)).await?;

The guard returns a clear error if the capability is missing.

Assembling the Scenario

build_failover_scenario (examples/openraft_kv/examples/src/lib.rs) puts workload, expectation, and capability together:

pub fn build_failover_scenario(
    run_duration: Duration,
    workload_timeout: Duration,
) -> anyhow::Result<Scenario<OpenRaftKvEnv, NodeControlCapability>> {
    Ok(OpenRaftKvScenarioBuilder::with_existing_openraft_kv_app(
        OpenRaftKvExistingClusterApp::nodes(3),
    )
    .enable_node_control()
    .with_run_duration(run_duration)
    .with_workload(OpenRaftKvClusterAccessible::new(3))
    .with_workload(
        OpenRaftKvFailoverWorkload::new()
            .first_batch(INITIAL_WRITE_BATCH)
            .second_batch(SECOND_WRITE_BATCH)
            .timeout(workload_timeout)
            .key_prefix(RAFT_KEY_PREFIX),
    )
    .with_expectation(
        OpenRaftKvConverges::new(TOTAL_WRITES)
            .timeout(run_duration)
            .key_prefix(RAFT_KEY_PREFIX),
    )
    .build()?)
}

The return type is Scenario<OpenRaftKvEnv, NodeControlCapability>, so only deployers that provide node control will accept it. Run it locally or on compose:

cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover
cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover

The openraft_kv_k8s_failover bin executes the same failover flow on Kubernetes, but uses ManualCluster imperatively (start_node per node, restart_node, wait_network_ready). See ManualCluster: Imperative Node Control.

Patterns

Restart-and-verify. The minimal chaos loop: write known data, restart_node, wait for readiness, verify the data survived. Restarts reuse the node’s existing working directory, so on-disk state survives them by default; use with_snapshot_dir to seed a restore from saved state; details are in Persistence, Snapshots, and Recovery Testing.

Leader failover. Restart the node that currently holds a distinguished role. The failover workload discovers the leader from observed cluster state instead of assuming a node index. Passing the old identity to the wait (different_from: Some(leader_id) above) verifies that leadership changed rather than accepting the old leader after it restarts.

Readiness waits after disruption. Wait before sending traffic after a restart. The example waits on observed application state (an agreed leader across all nodes) via an observation handle, which checks more than an HTTP readiness probe. In imperative flows, ManualCluster::wait_network_ready() covers transport-level readiness (the k8s bin calls it right after restart_node); inside a declarative workload, wait on observed state.

Pair chaos with continuous observation. A background observer polls every node through the disruption, so waits read stored snapshots and can report the last observation (timed out waiting for observed leader agreement ...; last observation: node=0 leader=None ...). A workload can also poll clients directly, but must then track the polling state itself.

Managed clusters get a minimum 30-second cooldown window after the workload phase before expectations run, allowing post-chaos state to settle; see Expectations and Evaluation.

See Also

• Scenario Capabilities — with_node_control and StartNodeOptions
• Continuous Observation — the observer used for recovery waits
• Persistence, Snapshots, and Recovery Testing — restart with retained state
• ManualCluster: Imperative Node Control — the imperative variant

Runtime Extensions

Runtime extensions are typed values prepared once per run (after nodes exist, before workloads start) and handed to workloads and expectations through the RunContext. The app layer and the observation runtime are built on them.

The Mechanism

The implementation is in testing-framework/core/src/scenario/runtime/extensions.rs and has three parts:

1. A factory registered on the builder. RuntimeExtensionFactory<E> is called by the deployer during preparation, when the deployment is resolved and node clients are available:

#[async_trait]
pub trait RuntimeExtensionFactory<E: Application>: Send + Sync {
    async fn prepare(
        &self,
        deployment: &E::Deployment,
        node_clients: NodeClients<E>,
    ) -> Result<PreparedRuntimeExtension, DynError>;
}

Register it with .with_runtime_extension_factory(Box::new(factory)). Factories run in registration order; any prepare error aborts the deployment.

The factory runs after the deployment is resolved and node clients exist. It prepares the value once so every workload and expectation can share it instead of rebuilding the same clients or polling loops.

2. A prepared value, optionally with cleanup. PreparedRuntimeExtension wraps one value of any Clone + Send + Sync + 'static type, with three constructors:

Cleanup guards are collected into the run’s cleanup chain and execute at teardown in reverse registration order; see Handle Ownership and Teardown for how the app layer separates those guards from handle access.

3. Typed retrieval from the context. The prepared values land in a type-indexed store inside RunContext:

// Somewhere in a workload or expectation:
let handle: MyHandle = ctx.require_extension::<MyHandle>()?;
// or, tolerating absence:
let maybe: Option<MyHandle> = ctx.extension::<MyHandle>();

extension::<T>() returns a clone of the stored value. Extension values should therefore be cheap to clone, typically by wrapping shared state in an Arc.

One Value Per Type

The store is keyed by TypeId. Registering two extensions that prepare the same type is a hard error at prepare time:

duplicate runtime extension type registered: <type name>

Because ctx.extension::<T>() returns one value by type, each type may be registered only once. To register several values with the same underlying shape, wrap them in distinct newtypes or, in the app layer, use named handles instead (see AppDeployment and DeployContext).

This rule is why a scenario allows only one with_app(...): the app layer registers its AppRuntime extension per call, and a second registration collides. Compose multiple applications inside one root AppDeployment instead; see AppHost and with_app.

Writing a Factory

A minimal factory that shares a client wrapper with all workloads:

use async_trait::async_trait;
use testing_framework_core::scenario::{
    DynError, NodeClients, PreparedRuntimeExtension, RuntimeExtensionFactory,
};

#[derive(Clone)]
struct FrontDoor(MyNodeClient);

struct FrontDoorFactory;

#[async_trait]
impl RuntimeExtensionFactory<MyEnv> for FrontDoorFactory {
    async fn prepare(
        &self,
        _deployment: &<MyEnv as Application>::Deployment,
        node_clients: NodeClients<MyEnv>,
    ) -> Result<PreparedRuntimeExtension, DynError> {
        let client = node_clients
            .snapshot()
            .first()
            .cloned()
            .ok_or("no nodes available")?;

        Ok(PreparedRuntimeExtension::new(FrontDoor(client)))
    }
}

// Registration:
let builder = builder.with_runtime_extension_factory(Box::new(FrontDoorFactory));

For an extension backed by a polling loop, spawn the task in prepare and return from_task(handle, join_handle). The runner aborts the task when the run tears down, so the loop cannot outlive the cluster. The observation runtime works this way. The pubsub example registers its feed this way (examples/pubsub/testing/integration/src/scenario.rs):

self.with_runtime_extension_factory(Box::new(PubSubTopicFeedFactory::new(topic)))

What Is Built on This

The following layers use runtime extension factories:

So when a workload calls ctx.require_app::<KvStoreCluster>() or ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>(), it is walking the same type-indexed store described above.

• App layer: AppHost and with_app
• Observation runtime: Continuous Observation

See Also

• Workloads and Concurrency — consuming extensions from workloads
• Continuous Observation — an extension backed by a polling task
• Handle Ownership and Teardown — cleanup ordering in depth

Continuous Observation

The observation runtime polls application state in the background and stores snapshots, histories, and event streams for workloads and expectations. It provides typed state inside the test process rather than external telemetry.

Shared Polling Runtime

Chaos and convergence tests repeatedly query state such as the current leader, whether every node has seen a key, or what changed after a restart. The observation runtime (testing-framework/core/src/observation/) runs one background polling task with shared error and staleness tracking. Workloads and expectations read the stored state.

Telemetry exports metrics, logs, and traces to external endpoints. Observation instead keeps typed application state inside the test and makes it synchronously queryable during the run.

The Observer Trait

An application defines how to poll and interpret its state; the runtime schedules the polling:

#[async_trait]
pub trait Observer: Send + Sync + 'static {
    type Source: Clone + Send + Sync + 'static;   // app-owned source handle
    type State: Send + Sync + 'static;            // retained materialized state
    type Snapshot: Clone + Send + Sync + 'static; // current view
    type Event: Clone + Send + Sync + 'static;    // delta emitted per cycle

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError>;

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError>;

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot;
}

Each cycle the runtime refreshes the source set, calls poll to advance State and collect delta Events, then derives a Snapshot from the state. ObservedSource<S> is just a name plus the app-owned source value (ObservedSource::new(name, source)), typically a node client.

Sources are re-queried every cycle through SourceProvider<S>:

#[async_trait]
pub trait SourceProvider<S>: Send + Sync + 'static {
    async fn sources(&self) -> Result<Vec<ObservedSource<S>>, DynError>;
}

StaticSourceProvider::new(sources) covers the common fixed-cluster case. A custom provider makes sources dynamic: it can return a different set each cycle, which lets observation stay correct across node restarts. SourceProviderFactory<E, S> builds the provider once node clients exist; any closure Fn(&E::Deployment, NodeClients<E>) -> Result<BoxedSourceProvider<S>, DynError> qualifies.

Plugging Into a Scenario

ObservationExtensionFactory<E, O> is a runtime extension factory: at prepare time it builds the source provider, starts the runtime, and stores the read handle in the RunContext (background task registered for abort-on-teardown via PreparedRuntimeExtension::from_task). The builder has convenience methods for it (CoreBuilderExt):

// Clonable observer:
builder.with_observer(MyObserver, my_source_provider_fn, ObservationConfig::default())
// Observer built lazily per run:
builder.with_observer_factory(|| MyObserver::new(), my_source_provider_fn, config)

ObservationConfig has two fields: interval (time between cycles, default 1 s, must be non-zero) and history_limit (retained non-empty event batches, default 64).

Outside scenarios, for example around a ManualCluster, start it directly: ObservationRuntime::start(provider, observer, config), then handle(), into_parts() (handle + JoinHandle), or abort(). Dropping the runtime aborts the task.

Reading: the ObservationHandle

Workloads and expectations retrieve the handle by type and read four things:

Snapshots vs batches vs events: a snapshot is the whole current view (cycle, observed_at, source_count, value); an event is one delta discovered during a cycle; a batch groups the events of one cycle. Cycles that produce no events produce no batch. history() and subscribe() only ever see non-empty batches, while latest_snapshot() is refreshed on every successful cycle.

Freshness and failures. On a failed cycle, the runtime records an ObservationFailure (with stage: SourceRefresh if source discovery failed, stage: Poll if the observer failed) and retains the last successful snapshot. The next successful cycle clears last_error. To check staleness, compare snapshot.cycle or observed_at across reads, and inspect last_error() when a wait times out; it usually names the source that stopped answering.

Worked Example: the OpenRaft Cluster Observer

examples/openraft_kv/testing/integration/src/observation.rs observes a Raft cluster. State and snapshot are the same type (the latest per-node states plus any per-source failures), and no delta events are emitted (Event = ()):

#[derive(Clone, Debug, Default)]
pub struct OpenRaftClusterObserver;

#[async_trait]
impl Observer for OpenRaftClusterObserver {
    type Source = OpenRaftKvClient;
    type State = OpenRaftClusterSnapshot;
    type Snapshot = OpenRaftClusterSnapshot;
    type Event = ();

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError> {
        Ok(capture_cluster_snapshot(sources).await)
    }

    async fn poll(
        &self,
        sources: &[ObservedSource<Self::Source>],
        state: &mut Self::State,
    ) -> Result<Vec<Self::Event>, DynError> {
        *state = capture_cluster_snapshot(sources).await;
        Ok(Vec::new())
    }

    fn snapshot(&self, state: &Self::State) -> Self::Snapshot {
        state.clone()
    }
}

capture_cluster_snapshot queries each source’s /state endpoint and records per-node errors as OpenRaftSourceFailure values instead of failing the cycle. A node restarting therefore appears as a named failure inside the snapshot. The snapshot type provides agreed_leader(different_from), all_voters_match(...), all_kv_match(...), and summary() for timeout messages.

Two source providers accompany it:

// Fixed: scenario runs, sources from the run's node clients.
pub fn openraft_cluster_source_provider(
    _deployment: &<OpenRaftKvEnv as Application>::Deployment,
    node_clients: NodeClients<OpenRaftKvEnv>,
) -> Result<BoxedSourceProvider<OpenRaftKvClient>, DynError> {
    Ok(Box::new(StaticSourceProvider::new(named_sources(node_clients.snapshot()))))
}

and OpenRaftManualClusterSourceProvider, a dynamic provider that re-resolves clients from a ManualCluster on every cycle so observation follows manual restarts. The scenario builder wires the fixed one in via with_observer(OpenRaftClusterObserver, openraft_cluster_source_provider, OpenRaftClusterObserver::config()).

The failover scenario waits on this observed state:

let observer = ctx.require_extension::<ObservationHandle<OpenRaftClusterObserver>>()?;
let leader = wait_for_observed_leader(&observer, timeout, None).await?;

External example: logos-blockchain’s BlockFeed is an adopter-side analog of this pattern: an observer in its own repository materializes block records, per-node head snapshots, and transaction statistics, using the same Observer/ObservationHandle mechanism.

See Also

• Runtime Extensions — the mechanism observation plugs into
• Chaos and Controlled Failure — observation-driven recovery waits
• Expectations and Evaluation — snapshot-based verdicts
• Telemetry and External Observability — the external counterpart

Telemetry and External Observability

Telemetry connects a scenario to external observability infrastructure such as Prometheus, an OTLP collector, and Grafana. It supports PromQL queries and external dashboards. For typed application state inside a test, use the observation runtime.

Observation and Telemetry

Telemetry endpoints are optional in every deployer. Without telemetry configuration, the scenario still runs and RunContext::telemetry() has no Prometheus backend.

Declaring Endpoints on the Builder

ObservabilityCapability (testing-framework/core/src/scenario/capabilities.rs) carries three optional URLs:

You populate it with ObservabilityBuilderExt (testing-framework/core/src/scenario/builder_ext.rs), which transitions a plain ScenarioBuilder<E> into an ObservabilityScenarioBuilder<E>, the capability-typed builder described in Scenario Capabilities:

use testing_framework_core::scenario::ObservabilityBuilderExt;

let scenario = ScenarioBuilder::with_deployment(topology)
    .with_metrics_query_url_str("http://127.0.0.1:9090")
    .with_metrics_otlp_ingest_url_str("http://127.0.0.1:4318")
    .with_run_duration(Duration::from_secs(60))
    .build()?;

Each endpoint has three setter flavors: with_..._url(Url), with_..._url_str(&str) (panics on an invalid URL), and try_with_..._url_str(&str) (returns BuilderInputError).

ObservabilityInputs: Capability Plus Environment

Deployers do not read the capability directly; they resolve an ObservabilityInputs (testing-framework/core/src/scenario/observability.rs) that merges two sources:

let env_inputs = ObservabilityInputs::from_env()?;
let cap_inputs = observability
    .observability_capability()          // via ObservabilityCapabilityProvider
    .map(ObservabilityInputs::from_capability)
    .unwrap_or_default();
let inputs = env_inputs.with_overrides(cap_inputs);

The compose and k8s orchestrators use this merge in testing-framework/deployers/{compose,k8s}/src/deployer/orchestrator.rs: environment values form the base, and any endpoint set on the scenario capability overrides the corresponding environment value.

This allows the environment to supply infrastructure-specific endpoints while the scenario can override individual URLs on the builder.

What from_env reads. Verified against the source, it reads exactly three environment variables, each parsed as a URL (empty or unset values are skipped; an unparsable value is an error):

ObservabilityInputs also offers from_capability(&cap), with_overrides(other) (field-wise, Some wins), and telemetry_handle(), which builds the Metrics value stored in the RunContext: Metrics::from_prometheus(url) when metrics_query_url is set, Metrics::empty() otherwise.

Deployer support today: compose and k8s resolve env + capability as above and wire the OTLP ingest URL into node configuration. The local deployer currently builds its runtime with Metrics::empty() and does not wire telemetry endpoints. See the Capability Matrix.

Querying Metrics in a Run

RunContext::telemetry() returns the Metrics handle. Backed by Prometheus it evaluates instant queries:

let telemetry = ctx.telemetry();
let values = telemetry.instant_values("up")?;          // all sample values
let total = telemetry.counter_value("requests_total")?; // summed counter

Without a configured metrics_query_url these calls return a MetricsError (“prometheus endpoint unavailable”). Expectations that assert on metrics therefore require a configured telemetry endpoint.

Telemetry queries depend on scrape intervals, exporter lag, and external infrastructure. Observation polls application state from the test process and reports failures by source. Correctness checks can use observation when they require current typed state; performance checks and post-run analysis can use telemetry.

A Local Stack for Development

To use a local Prometheus, OTLP collector, and Grafana stack, export the three environment variables above or set the URLs on the builder. The same scenario binary can then run with or without a metrics backend.

See Also

• Continuous Observation — test-visible state, the in-process counterpart
• Scenario Capabilities — how the observability capability is typed
• Capability Matrix — per-deployer telemetry support
• Environment Variables — the full audited env var list

Part IV — Uniform Clusters and Configuration

This part shows how to put your own node behind the framework and control how clusters are configured and driven.

It covers the uniform-cluster entry pattern in depth: implementing Application, describing topologies, generating per-node configuration, and driving nodes imperatively outside the scenario runtime.

• Implementing Application — the environment contract for your node
• Topology and Deployment Plans — describing cluster shape
• Ports, Peers, Node Config, and Readiness — per-node configuration mechanics
• Static Artifacts and cfgsync — typed app config to per-node artifacts to backend rendering
• Seeds and Reproducibility — deterministic deployments
• ManualCluster: Imperative Node Control — direct node lifecycle control
• Persistence, Snapshots, and Recovery Testing — state across restarts

Implementing Application

This chapter shows how to put your own node binary behind the framework so deployers can launch it as a uniform cluster.

The Application Trait

Every environment starts with Application (testing-framework/core/src/env.rs). It bundles the backend-agnostic types the scenario engine needs:

pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str;
}

The associated types and methods are:

Workloads and expectations use only these types, so the same scenario code can run against local processes, Compose containers, and Kubernetes services. Each backend still requires its corresponding deployment integration.

Application does not specify how nodes run. Each deployer adds a backend-specific integration trait.

Local Integration: Two Paths

The local deployer (testing-framework/deployers/local/src/env/mod.rs) offers two traits.

LocalBinaryApp covers apps that launch one binary per node, write one config file per node, and expose one HTTP API port. You implement five methods; a blanket implementation supplies LocalDeployerEnv:

Optional overrides: initial_local_port_names() (extra named ports reserved per node), readiness_endpoint_path(), readiness_probe() (HTTP GET or plain TCP), and wait_readiness_stable(nodes) for app-specific stabilization after the port probe succeeds.

LocalDeployerEnv exposes the deployer-facing hooks directly: build_node_config_from_template, build_initial_node_configs, build_launch_spec, node_endpoints, node_client, node_peer_port, local_process_spec_for_node (per-node binary selection for mixed-version clusters), and initial_persist_dir / initial_snapshot_dir (see Persistence). Implement it directly when LocalBinaryApp does not cover the application’s launch requirements.

graph LR
    A[Application] --> B[LocalBinaryApp]
    B -- blanket impl --> C[LocalDeployerEnv]
    C --> D["ProcessDeployer<E>"]

Worked Example: kvstore

The kvstore integration lives in examples/kvstore/testing/integration/src/. The environment type is an empty struct:

pub struct KvEnv;

#[async_trait]
impl Application for KvEnv {
    type Deployment = KvTopology;        // = ClusterTopology
    type NodeClient = KvHttpClient;
    type NodeConfig = KvNodeConfig;

    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError> {
        Ok(KvHttpClient::new(access.api_base_url()?))
    }

    fn node_readiness_path() -> &'static str {
        "/health/ready"
    }
}

Client construction. build_node_client receives NodeAccess, a host plus API port (and optional testing/named ports) discovered by the deployer, and wraps its base URL in the app’s HTTP client. The same function serves every backend: local processes, Compose containers, and K8s services all resolve to a NodeAccess.

Readiness path. node_readiness_path returns /health/ready. Deployers append it to http://<host>:<api-port> and poll until the node answers 2xx. See Ports, Peers, Node Config, and Readiness for the probe implementation.

The local side (local_env.rs) implements LocalBinaryApp:

impl LocalBinaryApp for KvEnv {
    fn initial_node_name_prefix() -> &'static str {
        "kv-node"
    }

    fn build_local_node_config_with_peers(
        _topology: &Self::Deployment,
        index: usize,
        ports: &LocalNodePorts,
        peers: &[LocalPeerNode],
        _peer_ports_by_name: &HashMap<String, u16>,
        _options: &StartNodeOptions<Self>,
        _template_config: Option<&KvNodeConfig>,
    ) -> Result<KvNodeConfig, DynError> {
        build_local_cluster_node_config::<Self>(index, ports, peers)
    }

    fn local_process_spec() -> LocalProcessSpec {
        LocalProcessSpec::new("KVSTORE_NODE_BIN")
            .with_binary_provider(kvstore_binary_provider())
            .with_rust_log("kvstore_node=info")
    }

    fn render_local_config(config: &KvNodeConfig) -> Result<Vec<u8>, DynError> {
        yaml_node_config(config)
    }

    fn http_api_port(config: &KvNodeConfig) -> u16 {
        config.http_port
    }
}

Config generation. kvstore delegates to build_local_cluster_node_config::<Self>, which works because KvEnv also implements ClusterNodeConfigApplication (app.rs): a backend-neutral hook that builds a NodeConfig from a ClusterNodeView (own index, host, ports) and ClusterPeerView list. Implementing that one trait gives kvstore local config generation and the static-artifact path used by Compose/K8s; see Static Artifacts and cfgsync.

Binary provider. kvstore_binary_provider() returns a FallbackBinaryProvider chain: first EnvBinaryProvider::new("KVSTORE_NODE_BIN") (use a prebuilt binary if the env var is set), then BuildBinaryProvider running cargo build -p kvstore-node --bin kvstore-node in the workspace root. This is why kvstore examples need no manual setup. Providers are covered in Binary Providers.

Launch and config rendering. At spawn time the framework renders the config with render_local_config, writes it as config.yaml into the node’s working directory, and launches <binary> --config config.yaml with the spec’s env vars. LocalProcessSpec supports different file names, positional config arguments, and extra args (see node-config.md).

Finally, lib.rs exports ready-made deployer aliases:

pub type KvLocalDeployer = testing_framework_runner_local::ProcessDeployer<KvEnv>;
pub type KvComposeDeployer = testing_framework_runner_compose::ComposeDeployer<KvEnv>;
pub type KvK8sDeployer = testing_framework_runner_k8s::K8sDeployer<KvEnv>;

Other Backends

The same KvEnv gains container support in two short files:

• compose_env.rs implements ComposeBinaryApp: a BinaryConfigNodeSpec naming the in-container binary path, config path, and exposed ports. See Compose Deployer.
• k8s_env.rs implements K8sBinaryApp: a BinaryConfigK8sSpec with release name, node-name prefix, binary and config paths, and service ports. See Kubernetes Deployer.

Both backends deliver generated configs through cfgsync rather than the local filesystem (Static Artifacts and cfgsync).

Implement traits only for the backends you use. An Application implementation plus LocalBinaryApp is sufficient for local scenarios.

Topology and Deployment Plans

This chapter explains how a scenario describes cluster shape and how a deployment provider turns that description into the concrete deployment the runner uses.

DeploymentDescriptor

The core contract is defined in testing-framework/core/src/topology/mod.rs:

pub trait DeploymentDescriptor: Send + Sync {
    fn node_count(&self) -> usize;
}

Every Application::Deployment implements it. The scenario engine itself only needs the node count; everything richer (per-node configs, ids, network layout) belongs to the app’s own deployment type and to the deployer that interprets it.

Built-in Topology Types

The topology module ships a few concrete building blocks. Verify against testing-framework/core/src/topology/:

Every example app that runs as a uniform cluster aliases ClusterTopology:

pub type KvTopology = testing_framework_core::topology::ClusterTopology;

let topology = KvTopology::new(3); // 3 nodes, indices 0..3

DeploymentPlan and NodePlan implement DeploymentDescriptor too, for apps whose deployment must carry a prebuilt per-node config (plans[i].general) instead of deriving configs at spawn time. TopologyShapeBuilder and DeploymentPlan are available building blocks; the in-repo example apps currently build on ClusterTopology directly.

Deployment Providers

A scenario does not have to hold a finished deployment. It holds a provider:

pub trait DeploymentProvider<D>: Send + Sync
where
    D: DeploymentDescriptor,
{
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

FixedDeploymentProvider<D> wraps a concrete deployment and clones it on every build, ignoring the seed. A custom provider can generate the deployment lazily: sized from the environment, randomized from the seed, or derived from an external inventory.

Feeding the Builder

ScenarioBuilder<E> accepts a deployment in three ways (core/src/scenario/definition/builder.rs):

Resolution happens once, inside build(): the builder calls provider.build(seed) and bakes the resulting deployment into the Scenario. Deployers and workloads then see a fixed descriptor for the rest of the run.

graph LR
    P[DeploymentProvider] -- "build(seed)" --> D[E::Deployment]
    S[with_deployment_seed] -. optional .-> P
    D --> SC["Scenario<E>"]
    SC --> R[Deployer / Runner]
    D:::cl
    SC:::sc
    classDef cl stroke:#4a90d9,stroke-width:2.5px;
    classDef sc stroke:#9b6dd6,stroke-width:2.5px;

The typical example flow, from kvstore (examples/kvstore/testing/integration/src/scenario.rs):

pub trait KvBuilderExt: Sized {
    fn deployment_with(f: impl FnOnce(KvTopology) -> KvTopology) -> Self;
}

impl KvBuilderExt for KvScenarioBuilder {
    fn deployment_with(f: impl FnOnce(KvTopology) -> KvTopology) -> Self {
        KvScenarioBuilder::with_deployment(f(KvTopology::new(3)))
    }
}

map_deployment_provider and with_deployment_provider exist on all three builder forms (ScenarioBuilder, NodeControlScenarioBuilder, ObservabilityScenarioBuilder) and on the shared CoreBuilderExt used by app-specific builders. Wrapper builders can forward them through that shared extension.

What the Deployment Does Downstream

• The local deployer reads node_count() and asks the environment to reserve ports and build one config per index; see Ports, Peers, Node Config, and Readiness.
• The container backends iterate indices to produce per-node static artifacts delivered through cfgsync; see Static Artifacts and cfgsync.
• ManualCluster treats the deployment as capacity: nodes are started on demand against the descriptor. See ManualCluster.

Ports, Peers, Node Config, and Readiness

This chapter describes how the local deployer allocates ports, wires peers, materializes per-node configs, and decides when a cluster is ready.

Port Allocation

All local ports come from the OS. preallocate_ports (in testing-framework/deployers/local/src/env/helpers.rs) binds 127.0.0.1:0, records the assigned port, and releases the listener. reserve_local_node_ports(count, names, label) does this for every node up front and returns one LocalNodePorts per node:

Named ports exist for apps that need more than one listener per node. Declare them via LocalBinaryApp::initial_local_port_names() (or LocalDeployerEnv::local_port_names()); the deployer reserves one port per name per node.

Ports are reserved by bind-and-release, so they are free at reservation time but are not deterministic across runs. See Seeds and Reproducibility.

Peer Wiring

Peer wiring is why up-front port reservation matters: because every node’s ports are reserved before any config is built, each node’s config can reference the real addresses of all its peers before a single process starts.

For each node index the deployer builds peer views of every other node:

• LocalPeerNode: index(), network_port(), http_address() / authority() (127.0.0.1:<port>).
• build_local_peer_nodes(peer_ports, self_index): full peer views, skipping self.
• build_indexed_http_peers(node_count, self_index, peer_ports, build_peer): map peers through your own constructor.

These flow into the app’s config hook together with the node’s own ports:

fn build_local_node_config_with_peers(
    topology: &Self::Deployment,
    index: usize,
    ports: &LocalNodePorts,
    peers: &[LocalPeerNode],
    peer_ports_by_name: &HashMap<String, u16>,
    options: &StartNodeOptions<Self>,
    template_config: Option<&Self::NodeConfig>,
) -> Result<Self::NodeConfig, DynError>;

For initial cluster startup the deployer calls this once per index with every other node as a peer (a full mesh view); the layout your nodes actually form is up to the config your app builds from those views. LocalBuildContext carries the same fields when you customize build_initial_node_configs on the full LocalDeployerEnv path. Apps that implement ClusterNodeConfigApplication can delegate the whole hook to build_local_cluster_node_config::<E>(index, ports, peers), the same abstraction the container backends reuse (see cfgsync).

Config Templates: LocalProcessSpec

LocalProcessSpec describes how one rendered config becomes a running process:

Rendering helpers: yaml_node_config (serialize to YAML bytes), text_node_config (already-rendered text), yaml_config_launch_spec / text_config_launch_spec / default_yaml_launch_spec (build a full LaunchSpec in one call). The resulting LaunchSpec lists the binary, the files to materialize, args, and env; the deployer writes the files into the node’s working directory and spawns the process there.

StartNodeOptions: Overrides at Start Time

Dynamically started nodes (node-control workloads and ManualCluster) accept StartNodeOptions<E>; the full field table lives in the ManualCluster chapter. The two config-shaping fields deserve care:

config_override replaces the complete generated config. config_patch (set via create_patch(|config| ...)) transforms the generated config, retaining framework-assigned ports and peers unless the callback changes them. A full override must provide every required port itself.

Where they are honored differs by path:

• Local dynamic starts (NodeManager::start_node_with): the framework builds the config through the env hooks (which receive the full options) and then applies config_patch itself. config_override and peers are visible to your build_local_node_config_with_peers implementation but are not interpreted centrally by the local path.
• Static-artifact path (used by the container backends through StaticNodeConfigProvider::build_node_artifacts_for_options, core/src/scenario/config.rs): the framework interprets everything: PeerSelection picks the peer set, then config_override replaces, then config_patch transforms, and the result is served as an override artifact.

PeerSelection variants (core/src/scenario/capabilities.rs):

Readiness

Per-node probe. The local deployer probes each node’s API port using LocalDeployerEnv::readiness_probe():

• LocalReadinessProbe::HttpGet { path } (default): GET http://127.0.0.1:<api-port><path> until it returns 2xx. The path defaults to Application::node_readiness_path() ("/" unless overridden; kvstore uses /health/ready).
• LocalReadinessProbe::Tcp: the port merely accepts TCP connections. Use for nodes without an HTTP surface.

Cluster requirement. HttpReadinessRequirement (core/src/scenario/runtime/readiness.rs) decides how many nodes must pass:

Set it on the scenario with ScenarioBuilder::with_http_readiness_requirement(requirement), or as part of a full DeploymentPolicy; see Readiness, Retry, and Artifact Preservation.

Waiting imperatively. ManualCluster (and LocalAppCluster) expose:

• wait_network_ready(): polls every started node’s API port with AllNodesReady.
• wait_node_ready(name): polls one node, honoring that node’s NodeRuntimeOptions::start_timeout if one was set via StartNodeOptions::with_start_timeout.

Default probe timeout is 60 seconds with a 200 ms poll interval; setting SLOW_TEST_ENV=true doubles timeouts. Timeouts fail with a message listing the endpoints that never answered.

App-specific stabilization. After the port probe succeeds during deployment, the local deployer calls wait_readiness_stable(nodes), a hook where an app can wait for cluster-level convergence (membership settled, leader elected) before workloads start. The default is a no-op.

sequenceDiagram
    participant D as Deployer
    participant N as Node process
    D->>N: spawn (config materialized in working dir)
    loop until 2xx or timeout
        D->>N: GET /health/ready
    end
    D->>D: requirement satisfied? (All/Any/AtLeast)
    D->>N: wait_readiness_stable(...)

Static Artifacts and cfgsync

This chapter covers how typed app configs become per-node file artifacts and how containerized backends deliver them to nodes that cannot see your filesystem.

Why cfgsync Exists

The local deployer writes each node’s rendered config into the node’s working directory. Compose and Kubernetes nodes run in containers without access to the host directory where the framework generated those configs. cfgsync transfers the generated per-node files at startup and when a node is restarted with overridden options.

cfgsync consists of a typed artifact model and an HTTP service. A node container starts, registers with the cfgsync server, fetches its artifact set, writes the files locally, and then starts the application. The same artifact types support runtime config overrides through replace_node_artifacts for dynamic node starts on Kubernetes.

sequenceDiagram
    participant R as Runner
    participant S as cfgsync server
    participant C as node container
    R->>S: render config + artifacts, start server
    C->>S: POST /register (identifier, ip, metadata)
    C->>S: POST /node
    S-->>C: NodeArtifactsPayload (files)
    C->>C: write files, exec node binary

The Crates

Protocol (cfgsync-core). NodeRegistration carries a stable identifier, an IPv4 address, and an opaque RegistrationPayload, adapter-owned JSON metadata the framework never interprets (with_metadata(&T) / from_json_str). The server answers /node with a NodeArtifactsPayload (schema version + files) or a structured error: MissingConfig (unknown node), NotReady (registered, artifacts pending), Internal. Client wraps the endpoints: register_node, fetch_node_config, fetch_node_config_status (→ ConfigFetchStatus::{Ready, NotReady, Missing}), and the administrative ReplaceNodeArtifactsRequest for swapping one node’s served files.

Sources. A server serves whatever its NodeConfigSource resolves:

• StaticConfigSource: an in-memory map of identifier → payload, built from payloads or from a NodeArtifactsBundle (per-node entries plus shared_files served to everyone). Registration succeeds only for known identifiers. Supports replace_node_artifacts.
• RegistrationConfigSource<M> (cfgsync-adapter) is registration-aware: it records registrations, snapshots them, and asks a materializer for artifacts on every resolve. Per-node overrides installed via replace_node_artifacts win over materialized files (shared files are still appended).

Materialization (cfgsync-adapter). The adapter contract is one trait:

pub trait RegistrationSnapshotMaterializer: Send + Sync {
    fn materialize_snapshot(
        &self,
        registrations: &RegistrationSnapshot,
    ) -> Result<MaterializationResult, DynCfgsyncError>;
}

RegistrationSnapshot is the current registration set, sorted by identifier for determinism. The result is NotReady (keep polling) or Ready(MaterializedArtifacts): per-node ArtifactSets keyed by identifier plus a shared set appended to every node (resolve(identifier) merges them). Wrappers: CachedSnapshotMaterializer caches results per snapshot, PersistingSnapshotMaterializer additionally pushes ready artifacts into a MaterializedArtifactsSink. A prebuilt MaterializedArtifacts value is itself a materializer that is always ready.

Runtime (cfgsync-runtime). ServerConfig { port, source } loads from YAML; ServerSource is static (serve precomputed artifacts, no registration required) or registration (require registration first). The runtime Client adds local materialization: OutputMap routes artifact paths to disk (OutputMap::under(root), config_and_shared(config_path, shared_dir), or explicit route(...)), and run_client_from_env drives the whole register-fetch-write loop from CFG_SERVER_ADDR, CFG_HOST_IDENTIFIER, CFG_HOST_IP, CFG_REGISTRATION_METADATA_JSON, and output paths (CFG_FILE_PATH, CFG_DEPLOYMENT_PATH). This is what runs inside node containers before the app binary starts.

From Typed Config to Artifacts

The boundary between your typed NodeConfig and cfgsync lives in testing-framework/core/src/cfgsync/mod.rs.

Apps that implement ClusterNodeConfigApplication (see Implementing Application) get StaticNodeConfigProvider for free: build a config for node i, rewrite it for backend hostnames (node-0.svc instead of 127.0.0.1), and serialize it. On top of that:

• build_static_artifacts::<E>(deployment, hostnames) produces a MaterializedArtifacts with one /config.yaml per node-<i> identifier. Hostname count must match the node count.
• render_and_write_registration_server::<E, _>(...) renders both the cfgsync server config YAML and the precomputed artifacts YAML to disk, with an enrich_artifacts hook for app-specific extras (shared files, additional per-node files).
• build_node_artifact_override::<E>(deployment, index, hostnames, options) builds the replacement artifact set for a node started with non-default StartNodeOptions; PeerSelection, config_override, and config_patch are interpreted for container backends here (see node-config.md).

The backends consume these directly: the Compose deployer calls write_registration_server_compose_configs to render the server config and artifacts into the generated stack directory before docker compose up (Compose Deployer); the K8s deployer exposes cfgsync_service, cfgsync_hostnames, and build_cfgsync_override_artifacts hooks on its environment trait and pushes override artifacts through replace_node_artifacts when its manual cluster starts nodes with options (Kubernetes Deployer).

graph LR
    A["ClusterNodeConfigApplication<br/>(typed NodeConfig)"] --> B["build_static_artifacts<br/>(MaterializedArtifacts)"]
    B --> C[cfgsync server]
    C --> D["cfgsync client in container<br/>(writes files)"]
    D --> E[node process]
    C:::pr
    D:::pr
    E:::pr
    classDef pr stroke:#e08a3c,stroke-width:2.5px;

Choosing a Shape

Precomputed (used by the framework deployers): all registrations are known up front, so the deployer materializes every artifact before the stack starts and serves them through a registration-kind source. The rendered cfgsync.artifacts.yaml remains in the stack directory for inspection.

Registration-driven: when artifacts depend on runtime facts (e.g. which IPs registered), implement RegistrationSnapshotMaterializer yourself and return NotReady until the snapshot is complete. The runnable examples in cfgsync/runtime/examples/ (minimal_cfgsync.rs, precomputed_registration_cfgsync.rs, wait_for_registrations_cfgsync.rs) show both shapes end to end.

The framework deployers currently use the precomputed path. The registration-driven materializer is public API for integrations whose per-node configs cannot be finalized before nodes start.

Seeds and Reproducibility

DeploymentSeed controls only part of a run’s variability. This chapter lists what it seeds, what it does not, and what that means for reproducing a run.

DeploymentSeed

DeploymentSeed (testing-framework/core/src/topology/mod.rs) is a 32-byte value:

let seed = DeploymentSeed::new([7u8; 32]);
let bytes: &[u8; 32] = seed.bytes();

The seed exists so generated deployments can be reproduced: record it when a run fails, and the same seed makes the provider return the identical deployment.

You attach it to a scenario with with_deployment_seed:

let scenario = ScenarioBuilder::<KvEnv>::new(provider)
    .with_deployment_seed(DeploymentSeed::new([7u8; 32]))
    .with_run_duration(Duration::from_secs(30))
    .build()?;

The seed has exactly one consumer: when build() resolves the deployment, it calls the deployment provider with it:

pub trait DeploymentProvider<D>: Send + Sync {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

A provider that generates topologies (random shapes, sampled node parameters, derived node ids) should draw all of its randomness from the seed, so the same seed always yields the same deployment. See Topology and Deployment Plans for how providers feed the builder.

What Is Actually Seeded

The current behavior is:

No in-repo deployment provider currently consumes the seed: FixedDeploymentProvider is the only provider shipped, and the example apps all use concrete ClusterTopology values. DeploymentSeed is available to custom generating providers; setting a seed on a fixed deployment has no effect.

Writing a Seeded Provider

A provider that wants reproducible generation reads all of its variability from the seed bytes:

use testing_framework_core::topology::{
    ClusterTopology, DeploymentProvider, DeploymentSeed, DynTopologyError,
};

struct SizedFromSeed {
    min_nodes: usize,
    max_nodes: usize,
}

impl DeploymentProvider<ClusterTopology> for SizedFromSeed {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<ClusterTopology, DynTopologyError> {
        let first = seed.map_or(0, |seed| seed.bytes()[0] as usize);
        let span = self.max_nodes - self.min_nodes + 1;
        Ok(ClusterTopology::new(self.min_nodes + first % span))
    }
}

let scenario = ScenarioBuilder::<KvEnv>::new(Box::new(SizedFromSeed {
        min_nodes: 3,
        max_nodes: 7,
    }))
    .with_deployment_seed(DeploymentSeed::new([7u8; 32]))
    .with_run_duration(Duration::from_secs(30))
    .build()?;

The same seed always produces the same cluster size; omitting the seed uses the provider’s default (seed is None). A generating provider can feed the 32 bytes into a seeded RNG and derive node parameters or NodePlan ids from it.

Practical Reproducibility

• If your provider is seeded, record the seed alongside failures and replay with with_deployment_seed to get the identical deployment.
• Everything downstream of the deployment (ports, PIDs, timing) still varies run to run. Determinism ends at the descriptor; treat expectations accordingly.
• For state-level reproduction (replaying a node from captured state rather than regenerating a topology), use snapshot directories instead; see Persistence, Snapshots, and Recovery Testing.

ManualCluster: Imperative Node Control

ManualCluster provides imperative node lifecycle control. Your code starts, stops, and restarts nodes directly without using the scenario runner.

When to Use It

Use ManualCluster when orchestration lives outside the scenario runtime. Scenarios can also start and restart nodes from workloads by requesting with_node_control(); see Scenario Capabilities and Chaos and Controlled Failure.

• Step-driven flows: an external driver decides when each node starts and what happens next.
• BDD harnesses: Gherkin steps map naturally onto imperative start/stop/wait calls.
• Exploratory debugging: poke at a live cluster from a main function without writing workloads or expectations.

There are no workloads, expectations, or RunContext; you call methods and assert with your own code.

Creating a Cluster

Two equivalent entry points on the local backend (testing-framework/deployers/local/src/manual/mod.rs):

use testing_framework_runner_local::{ManualCluster, ProcessDeployer};

// Directly from a deployment descriptor…
let cluster = ManualCluster::<KvEnv>::from_topology(KvTopology::new(3));

// …or via the deployer
let deployer = ProcessDeployer::<KvEnv>::new();
let cluster = deployer.manual_cluster_from_descriptors(KvTopology::new(3));

The descriptor defines capacity and indexing, not initial state: no processes exist until you call start_node. E must implement LocalDeployerEnv (see Implementing Application).

Naming: requested names are normalized to a node- prefix: start_node("a") registers node-a; names already starting with node- pass through; an empty name becomes node-<index>. Each started node needs a fresh name; reusing a registered name is an error.

API

ManualCluster also implements the core NodeControlHandle<E> and ClusterWaitHandle<E> traits, so it can stand behind code written against those abstractions. An app-layer child cluster exposes the same common operations through ClusterHandle<E>, without exposing the backend-specific ManualCluster object.

StartNodeOptions

The full options struct (core/src/scenario/capabilities.rs):

restart_node_with accepts only args and runtime. Passing peers, config_override, config_patch, persist_dir, or snapshot_dir to a restart returns an InvalidArgument error, because a restart reuses the node’s existing config and working directory. To change those, stop the node and start a new one.

Example: Convergence Under Restart

Adapted from the in-repo example cargo run -p kvstore-examples --bin kvstore_k8s_manual_convergence (examples/kvstore/examples/src/bin/k8s_manual_convergence.rs):

let deployer = KvK8sDeployer::new();
let cluster = deployer
    .manual_cluster_from_descriptors(KvTopology::new(3))
    .await?;

let node0 = cluster.start_node("node-0").await?.client;
let node1 = cluster.start_node("node-1").await?.client;
let node2 = cluster.start_node("node-2").await?.client;
cluster.wait_network_ready().await?;

write_keys(&node0, "kv-manual", 12).await?;
wait_for_convergence(&[node0.clone(), node1.clone(), node2.clone()], "kv-manual", 12).await?;

cluster.restart_node("node-2").await?;
cluster.wait_network_ready().await?;

let node2 = cluster.node_client("node-2").expect("client after restart");
wait_for_convergence(&[node0, node1, node2], "kv-manual", 12).await?;

cluster.stop_all();

The driver determines which nodes exist, when writes happen, what convergence means, and when to inject the restart. write_keys and wait_for_convergence are plain functions over the application’s HTTP client.

That example runs on Kubernetes: the Kubernetes deployer supplies a manual cluster with the same method surface (manual_cluster_from_descriptors there is async and fallible because it must install the stack first). The local ManualCluster documented in this chapter starts processes directly and needs no external infrastructure.

Lifecycle and Cleanup

Dropping the ManualCluster calls stop_all(): every child process is killed and waited on. Node working directories are temporary and removed with the processes unless retained. Set TF_KEEP_LOGS=1 (or true/yes) to keep them for inspection, and see Persistence for deliberate state retention.

External example: logos-blockchain’s cucumber suite drives ManualCluster from Gherkin steps in its own repository, including dependency-ordered starts, targeted restarts, snapshot-on-stop, and restore-from-snapshot.

Persistence, Snapshots, and Recovery Testing

This chapter explains how node working directories behave, what persist_dir and snapshot_dir actually do, and how to build stop/restore recovery tests on top of them.

The Working Directory

Every locally spawned node runs inside a framework-created directory (testing-framework/deployers/local/src/process.rs):

• Launch files (the rendered config) are written into it before spawn, and the process starts with it as its current directory. A node that writes relative paths (a database under ./db, logs under ./logs) keeps all its state there.
• By default the directory is a random-named temp directory created in the current working directory of the test process, and it is deleted when the node is dropped.
• Deletion is skipped when the owning thread is panicking, when the node was spawned with keep_tempdir, when TF_KEEP_LOGS=1 is set, or when the deployment policy sets cleanup_policy.preserve_artifacts (see Readiness, Retry, and Artifact Preservation).

restart (used by ManualCluster::restart_node and LocalProcessHandle::restart) kills the child and respawns it in the same directory with the same launch spec. Launch files are rewritten; everything else is untouched, so state in the working directory survives the restart.

persist_dir: a Predictable Location

persist_dir does not reuse the given path as-is. Verified semantics from create_tempdir:

• The working directory is created as <basename>_<random-suffix> inside the parent of the path you pass. with_persist_dir("/tmp/kv-run/node-0") yields a working directory like /tmp/kv-run/node-0_a1B2c3/. The parent is created if missing.
• Nothing is copied into it; it starts empty apart from launch files.
• It is still a managed temp directory: deleted on drop unless one of the retention switches above applies.

Use persist_dir when a test (or a human) must find the node’s state. Pair it with TF_KEEP_LOGS=1 or preserve_artifacts to keep the directory after the run, then feed it back in as a snapshot later.

snapshot_dir: Seeding State at Start

snapshot_dir copies saved state into the fresh working directory before the process spawns. This is the restore half of a recovery test: a fresh node starts from state captured in an earlier run instead of an empty directory. Verified semantics from copy_snapshot_dir:

• The directory you pass is copied as a subdirectory of the working directory, named after its final path component, with overwrite enabled. with_snapshot_dir("/snapshots/run1/db") produces <workdir>/db/....
• Consequently, the snapshot source’s basename must match the relative path the node expects. If your node reads ./db, snapshot a directory literally named db.
• The copy happens once, at spawn. Restarts do not re-apply it.
• A failed copy fails the spawn (ProcessSpawnError::Snapshot).

The framework copies the supplied directory byte-for-byte without interpreting its contents. The caller determines what constitutes a consistent snapshot, including which directories to copy, whether the node must be stopped first, and whether its on-disk state is crash-consistent.

Where the Options Live

Per dynamic node: StartNodeOptions (full table in ManualCluster):

let node = cluster.start_node_with(
    "restored",
    StartNodeOptions::default()
        .with_snapshot_dir(PathBuf::from("/snapshots/run1/db"))
        .with_start_timeout(Duration::from_secs(90)),
).await?;

Note restart_node_with rejects persist_dir/snapshot_dir overrides, since restarts keep the existing directory. Start a new node to restore from a snapshot.

Per initial node: LocalDeployerEnv::initial_persist_dir(topology, node_name, index) and initial_snapshot_dir(...) (default None). Override these to mount state under the whole initial cluster, e.g. restore every node of a 3-node cluster from a saved dataset before the scenario begins.

Per composed process: LocalProcessApp in the app layer (testing-framework/app/src/process.rs) exposes the same three switches for one-binary apps:

Its LocalProcessHandle offers working_dir(), restart(), stop(), is_running(), pid(), and keep_tempdir(); the process stops when the last handle clone drops (see One Binary: LocalProcessApp).

Recovery-Testing Patterns

Restart in place. State persists because the node reuses its working directory.

write_data(&client).await?;
cluster.restart_node("node-1").await?;
cluster.wait_node_ready("node-1").await?;
assert_data_recovered(&cluster.node_client("node-1").unwrap()).await?;

Stop, snapshot, restore. Full recovery drill via ManualCluster:

// 1. Run a node whose working dir you can locate.
let node = cluster.start_node_with(
    "primary",
    StartNodeOptions::default().with_persist_dir(PathBuf::from("/tmp/kv-run/primary")),
).await?;
write_data(&node.client).await?;

// 2. Stop it, then copy its state out yourself (caller-owned step):
cluster.stop_node("node-primary").await?;
//    e.g. cp -r /tmp/kv-run/primary_*/db /snapshots/case1/db

// 3. Start a fresh node seeded from the snapshot.
let restored = cluster.start_node_with(
    "restored",
    StartNodeOptions::default().with_snapshot_dir(PathBuf::from("/snapshots/case1/db")),
).await?;
cluster.wait_node_ready("node-restored").await?;
assert_data_recovered(&restored.client).await?;

Step 2 is caller-owned because the framework neither snapshots on stop nor knows which files constitute application state. Set TF_KEEP_LOGS=1 so stopped nodes’ directories remain available for copying.

Config continuity. The first node started with a snapshot_dir has its generated config recorded as a template, and that template is passed to the config-build hooks of later dynamic starts (the template_config parameter). An env that honors it can keep restored nodes consistent with the configs their state was produced under.

Cross-run state. Combine persist_dir (findable location) with retention (TF_KEEP_LOGS / preserve_artifacts), archive the directory after run A, and hand it to run B via snapshot_dir or the initial_snapshot_dir hook. Upgrade tests, long-lived-ledger tests, and crash-recovery matrices all reduce to this loop.

Part V — Deployers and Sources

This part covers where scenarios run and where their nodes come from.

Uniform scenarios deploy to local processes, Docker Compose, or Kubernetes; scenarios can also attach to clusters you already operate. The app layer uses the same cluster request and handle model, with local as its implemented provisioning backend today.

• Capability Matrix — feature support per deployer
• Local Deployer — processes on your machine
• Compose Deployer — containerized clusters
• Kubernetes Deployer — Helm releases in isolated namespaces
• Shared Cluster Provisioning — one request and handle model across cluster sources
• Existing and External Clusters — attaching to live systems
• Binary Providers — resolving node binaries: path, env, build, download
• Readiness, Retry, and Artifact Preservation — deployment policy

Capability Matrix

This page records what each deployer backend currently supports, based on the deployer implementations.

The framework ships three deployers: ProcessDeployer (local processes), ComposeDeployer (Docker Compose), and K8sDeployer (Kubernetes/Helm). All three drive the same scenario runtime; they differ in where nodes run and which capabilities they wire into it.

Deployer and App-System Coverage

testing-framework-app is not a fourth deployer. It is a scenario runtime extension for composing typed application units. Its built-in process and cluster adapters currently use the local deployer primitives.

Legend: Yes = implemented, Partial = implemented with the stated limits, No = no implementation in that subsystem, and Inherited = the app layer uses the enclosing scenario/deployer behavior.

Example Application Coverage

This table records concrete adapters and runnable example binaries, rather than what the generic framework could theoretically support.

Row-by-Row

Uniform managed scenarios. All three deployers implement the Deployer trait for scenarios built with ScenarioBuilder<E> over a topology: deployer.deploy(&scenario).await returns a Runner<E>. This is the common path shown in the Local, Compose, and Kubernetes chapters.

Node control. The local deployer implements Deployer<E, NodeControlCapability> and backs it with a NodeManager that can start, stop, and restart node processes, including StartNodeOptions (peer selection, config overrides, persist/snapshot dirs). The compose deployer wires a ComposeNodeControl handle that supports restart_node via docker compose restart; in attached (existing-cluster) mode it also supports stop_node via docker container stop. The k8s deployer wires a K8sNodeControl handle into managed scenario deployments that supports start_node, stop_node, restart_node, and wait_node_ready by scaling the per-node deployments; only default start options are accepted (config overrides, persist/snapshot dirs, extra args, and timeout overrides are rejected). It works over both direct NodePorts and kubectl port-forward fallback — after a restart or start the node’s forwards are respawned on their original local ports, so existing clients keep working. Attached (existing-cluster) mode rejects node control. Config-override lifecycle control on Kubernetes goes through the k8s ManualCluster (see Kubernetes Deployer and ManualCluster).

Observability / telemetry inputs. Compose and k8s resolve ObservabilityInputs from LOGOS_BLOCKCHAIN_METRICS_QUERY_URL / LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL / LOGOS_BLOCKCHAIN_GRAFANA_URL env vars merged with the scenario’s ObservabilityCapability, pass the OTLP ingest URL into workspace preparation, and build the run’s Metrics telemetry handle from the query URL. The local orchestrator constructs its runtime with Metrics::empty() and never resolves observability inputs. See Telemetry and External Observability.

Attach / existing clusters. with_existing_cluster(...) switches the scenario to ClusterMode::ExistingCluster. Compose accepts descriptors built with ExistingCluster::for_compose_project / for_compose_services; k8s accepts for_k8s_selector / for_k8s_selector_in_namespace. The local deployer explicitly rejects existing-cluster mode with a source-orchestration error. Details in Existing and External Clusters.

External nodes. All three deployers resolve with_external_node(s) sources into node clients through Application::external_node_client. The local deployer additionally falls back to a generic endpoint parser (build_external_client) when the application does not override that hook.

App layer / AppHost composition. The app layer is local-only today: AppHostLocalDeployer is a type alias for ProcessDeployer<AppHostEnv>. There is no compose or k8s AppHost deployer. See Backend Scope.

Binary providers. Binary resolution (PathBinaryProvider, EnvBinaryProvider, BuildBinaryProvider, DownloadBinaryProvider, FallbackBinaryProvider) lives in the local deployer crate and feeds LocalProcessSpec. Compose and k8s nodes run container images instead, so image selection happens through descriptor specs and env-var overrides, not binary providers. See Binary Providers.

cfgsync artifacts. The compose deployer writes a cfgsync.yaml into its workspace and can launch a Docker-backed cfgsync config server sidecar (ComposeConfigServerMode::Docker); the k8s deployer supports cfgsync-backed config overrides in manual-cluster flows and cfgsync-rendered bootstrap assets in chart values. The local deployer materializes rendered config files directly into each node’s working directory with no cfgsync involvement. See Static Artifacts and cfgsync.

Backend Selection

The local backend runs node processes directly and provides full node control. It requires no infrastructure beyond the node binary, which a binary provider can build.

Use Compose for container images, container networking, or telemetry endpoints. Use Kubernetes to exercise charts, NodePort or port-forward access paths, and cluster infrastructure. Attach to an already-running stack when the cluster outlives the test (see Existing and External Clusters).

Readiness gating, deploy retries, and artifact preservation are controlled uniformly through DeploymentPolicy; see Readiness, Retry, and Artifact Preservation.

Local Deployer

ProcessDeployer runs every node as a local OS process. It is the default backend.

The local deployer lives in the testing-framework-runner-local crate. It requires no Docker daemon and no cluster: it resolves a node binary, writes each node’s config into a private working directory, spawns the processes, probes readiness, and hands the running cluster to the scenario runner.

use kvstore_runtime_ext::KvLocalDeployer; // = ProcessDeployer<KvEnv>
use testing_framework_core::scenario::Deployer;

let deployer = KvLocalDeployer::default();
let runner = deployer.deploy(&scenario).await?;
runner.run(&mut scenario).await?;

Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_basic_convergence. No manual binary setup is needed, because kvstore’s fallback provider chain builds the node binary on first use (see Binary Providers).

What deploy Does

For a managed scenario, ProcessDeployer::deploy:

1. Validates the cluster mode: the local deployer rejects ClusterMode::ExistingCluster (attach is a compose/k8s feature, see Existing and External Clusters).
2. Builds the source orchestration plan and spawns one ProcessNode per topology entry.
3. Probes readiness and retries the whole spawn on failure (see below).
4. Merges external node clients into the managed set.
5. Assembles the runtime and returns a Runner<E> whose cleanup guard owns the node processes.

The main runtime types are:

• ProcessDeployer<E>: the deployer. E implements LocalDeployerEnv (full-control hooks) or the compact LocalBinaryApp trait (one binary + one config file + one HTTP port per node).
• LaunchSpec: the launch plan for one process: binary path, files to materialize, CLI args, env vars.
• ProcessNode: a spawned child process plus its tempdir, endpoints, and typed client.

Working Directories and Logs

Each node gets its own temporary working directory, created under the current directory (or under a caller-supplied persist path). Config files and any other LaunchFile entries are written there before spawn, and the process starts with that directory as its cwd.

Node stdout and stderr are inherited from the test process, so node logs interleave with your test output; control verbosity with the RUST_LOG value configured on the app’s LocalProcessSpec (for example .with_rust_log("kvstore_node=info")).

On drop, each ProcessNode kills its child and removes the tempdir. Two things preserve working directories instead of deleting them:

• DeploymentPolicy with cleanup_policy.preserve_artifacts = true (see Readiness, Retry, and Artifact Preservation), or the TF_KEEP_LOGS=1 env var.
• A panicking test thread, in which case directories are kept automatically for inspection.

Nodes started with a persist_dir or seeded from a snapshot_dir (via StartNodeOptions) copy or place state accordingly before spawn (see Persistence, Snapshots, and Recovery Testing).

Ports

The deployer reserves real OS ports up front: allocate_available_port() binds an ephemeral listener and releases it, and reserve_local_node_ports reserves the network port plus any app-named extra ports for each node. Endpoints are surfaced as NodeEndpoints (an API socket address plus named extra ports), from which the app builds its typed NodeClient.

Readiness and Retry

Readiness is governed by the scenario’s DeploymentPolicy combined with the deployer’s own switch:

The probe shape comes from the environment: LocalReadinessProbe::HttpGet { path } (default, using Application::node_readiness_path()) or LocalReadinessProbe::Tcp. If spawn or readiness fails, all nodes from that attempt are dropped and the entire cluster is respawned with exponential backoff and jitter, up to the retry budget.

Node Control

The local deployer supports the complete node-control surface. Building the scenario with with_node_control() deploys through Deployer<E, NodeControlCapability>, which wraps the spawned nodes in a NodeManager. Workloads can then start, stop, and restart nodes by name, with full StartNodeOptions support (peer selection, config overrides and patches, persist and snapshot directories, extra args, start timeouts). The openraft_kv failover scenario uses this path:

cargo run -p openraft-kv-examples --bin openraft_kv_basic_failover

See Scenario Capabilities for the capability-gated builder.

Manual Clusters

For orchestration outside the scenario runner, such as Cucumber steps or another test harness, the deployer provides an imperative cluster:

let deployer = ProcessDeployer::<KvEnv>::new();
let cluster = deployer.manual_cluster_from_descriptors(descriptors);

cluster.start_node("node-0").await?;
cluster.wait_network_ready().await?;
cluster.stop_all();

ManualCluster exposes start_node(_with), stop_node, restart_node(_with), wait_node_ready, wait_network_ready, node_client, node_pid, node_clients, and add_external_sources / add_external_clients. It is covered in depth in ManualCluster: Imperative Node Control.

Binary Resolution

Every local node needs an executable. LocalProcessSpec::new("MY_NODE_BIN") defaults to an env-var provider; with_binary_provider swaps in any BinaryProvider, including fallback chains that try an env override first and build with Cargo otherwise. Resolution is cached per process and locked across processes. Full detail in Binary Providers.

The local deployer supports external node sources (with_external_node) but not attached existing clusters. If Application::external_node_client is not implemented, it falls back to parsing the endpoint (http://host:port) and building a client from the resolved socket address.

Compose Deployer

ComposeDeployer runs each node as a Docker Compose service generated from your deployment descriptor.

The compose deployer lives in the testing-framework-runner-compose crate. It generates a compose file per run, brings the stack up, discovers the host ports Docker assigned, probes readiness, and hands control to the scenario runner. It requires a running Docker daemon; otherwise deployment returns ComposeRunnerError::DockerUnavailable.

use kvstore_runtime_ext::KvComposeDeployer; // = ComposeDeployer<KvEnv>
use testing_framework_core::scenario::Deployer;
use testing_framework_runner_compose::ComposeRunnerError;

let deployer = KvComposeDeployer::new();
let runner = match deployer.deploy(&scenario).await {
    Ok(runner) => runner,
    Err(ComposeRunnerError::DockerUnavailable) => return Ok(()), // skip without Docker
    Err(error) => return Err(error.into()),
};
runner.run(&mut scenario).await?;

Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_compose_convergence.

Deployment Pipeline

flowchart LR
    A[Workspace<br/>tempdir] --> B[Write configs<br/>+ cfgsync.yaml]
    B --> C[Render<br/>compose.generated.yml]
    C --> D[docker compose<br/>create + up]
    D --> E[Port discovery<br/>docker compose port]
    E --> F[Readiness<br/>probes]
    F --> G[Node clients<br/>+ Runner]

1. Workspace. A temporary ComposeWorkspace is created; the app’s ComposeDeployEnv::prepare_compose_configs writes per-node config files (for ComposeBinaryApp environments, one static config per node under stack/configs/, rewritten for service hostnames node-0, node-1, …).
2. cfgsync. If the environment enables ComposeConfigServerMode::Docker, a cfgsync config server container is started on an ephemeral port and the deployer waits for it to accept TCP connections before proceeding. The default mode is Disabled. See Static Artifacts and cfgsync.
3. Compose file. The env’s compose_descriptor (image, entrypoint, volumes, ports, environment, optional platform per service) is rendered through the Tera template at testing-framework/deployers/compose/assets/docker-compose.yml.tera into compose.generated.yml. The template is resolved relative to the repository root (CARGO_WORKSPACE_DIR override respected). Required images are checked with docker image inspect up front. The deployer never builds or pulls them; a missing image fails the deploy with MissingImage.
4. Bring-up. docker compose create and docker compose up run under a unique project name (compose-stack-<uuid>). On failure, container logs are dumped before cleanup.
5. Ports. Container ports map to ephemeral host ports; the deployer resolves each with docker compose port and records them as NodeHostPorts { api, testing }. The host defaults to 127.0.0.1 and can be overridden with COMPOSE_RUNNER_HOST.
6. Readiness. Per the env’s ComposeReadinessProbe: HTTP GET against Application::node_readiness_path() on each mapped API port, or raw TCP reachability. Gated by DeploymentPolicy.readiness_enabled and the deployer’s own with_readiness(bool) switch; when disabled, the stack gets a short fixed grace period instead. See Readiness, Retry, and Artifact Preservation.
7. Clients. build_node_client runs against the discovered host/port pairs, producing the scenario’s typed node clients.

Node Control

With with_node_control() on the builder, the deployer installs a ComposeNodeControl handle bound to the generated compose file and project. It supports restart only: restart_node(name) shells out to docker compose restart <service>. Start and stop of individual services are not wired for managed compose scenarios. The openraft_kv failover scenario runs on this backend: cargo run -p openraft-kv-examples --bin openraft_kv_compose_failover.

Attaching to an Existing Stack

The compose deployer fully supports existing-cluster mode. A scenario built with with_existing_cluster(ExistingCluster::for_compose_project("my-project")) skips workspace generation entirely: services are discovered from the running project (or taken from for_compose_services), each container’s labeled API port is inspected, and clients are built through Application::external_node_client. In this mode node control gains stop_node in addition to restart_node, implemented with docker container stop / docker container restart against discovered container IDs.

deploy_with_metadata returns ComposeDeploymentMetadata alongside the runner; its existing_cluster() / IntoExistingCluster impl lets a later scenario attach to the stack this one deployed. See Existing and External Clusters.

Observability

Compose resolves ObservabilityInputs by merging LOGOS_BLOCKCHAIN_METRICS_QUERY_URL, LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL, and LOGOS_BLOCKCHAIN_GRAFANA_URL env vars with the scenario’s observability capability (capability values win). The OTLP ingest URL is passed into config preparation so node configs can point at your collector; the metrics query URL becomes the run’s Prometheus-backed Metrics handle. Setting TESTNET_PRINT_ENDPOINTS prints Prometheus/Grafana endpoints and per-node pprof profile URLs to stdout. See Telemetry and External Observability.

Cleanup

The runner’s cleanup guard runs docker compose down, shuts down the cfgsync container if one was started, and removes the workspace. Setting COMPOSE_RUNNER_PRESERVE (or TESTNET_RUNNER_PRESERVE) keeps the stack running and persists the workspace directory for post-mortem inspection; the preserved path is logged.

Requirements recap:

Kubernetes Deployer

K8sDeployer installs each scenario as a Helm release in a throwaway namespace and reaches nodes through NodePorts or port-forwards.

The k8s deployer lives in the testing-framework-runner-k8s crate. It talks to whatever cluster your current kubeconfig context points at (kube::Client::try_default()), installs a Helm release, waits for the workloads, and builds node clients against externally reachable ports.

use kvstore_runtime_ext::KvK8sDeployer; // = K8sDeployer<KvEnv>
use testing_framework_core::scenario::Deployer;
use testing_framework_runner_k8s::K8sRunnerError;

let deployer = KvK8sDeployer::new();
let runner = match deployer.deploy(&scenario).await {
    Ok(runner) => runner,
    Err(K8sRunnerError::ClientInit { .. }) => return Ok(()), // no cluster available
    Err(error) => return Err(error.into()),
};
runner.run(&mut scenario).await?;

Run the demonstration binary with cargo run -p kvstore-examples --bin kvstore_k8s_convergence.

Charts and Values

The environment trait K8sDeployEnv produces installable assets via prepare_assets, returning a PreparedK8sStack. Two asset shapes exist:

• Generated single-template charts. Apps implementing K8sBinaryApp get the standard shape for free: render_binary_config_node_manifest renders one ConfigMap (the serialized node config), one Deployment (single replica, --config arg, config mounted from the ConfigMap), and one NodePort Service per node, then render_manifest_chart_assets wraps them in a minimal chart (RenderedHelmChartAssets).
• Real chart directories. NodeRuntimeSpec builds RunnerChartValues (node image, pull policy, fullname override, asset mount layout, node group, optional shared bootstrap service with cfgsync configs and start scripts) and a HelmReleaseBundle with --set values and --set-file entries for start scripts and bootstrap configs. RunnerAssetLayout fixes where bootstrap configs and runner scripts land inside the chart’s mount path.

Node images are resolved from env vars: for a conventional BinaryConfigK8sSpec the primary override is <APP>_K8S_IMAGE, the fallback <APP>_IMAGE, and the default <binary-name>:local with imagePullPolicy: IfNotPresent.

Each run installs into fresh identifiers: namespace tf-testnet-<timestamp>-<pid>, release tf-runner (override via K8sDeployEnv::cluster_identifiers).

Lifecycle Waits

After helm install, the deployer waits in stages:

1. Deployment readiness: each node Deployment must report ready replicas (timeout K8S_RUNNER_DEPLOYMENT_TIMEOUT_SECS, default 180 s).
2. Port discovery: each node Service must have allocated NodePorts for the API and auxiliary ports declared by collect_port_specs.
3. HTTP readiness: nodes are probed over their NodePorts at node_readiness_path(). The probe host is K8S_RUNNER_NODE_HOST if set, else KUBERNETES_SERVICE_HOST, else 127.0.0.1. If NodePort probing fails (common when the cluster’s node IPs are not routable from the runner), the deployer transparently falls back to kubectl port-forward per service and probes over 127.0.0.1.
4. Policy-gated cluster readiness: a final probe pass controlled by DeploymentPolicy.readiness_enabled / readiness_requirement and the deployer’s with_readiness(bool) switch. See Readiness, Retry, and Artifact Preservation.

HTTP wait tuning: K8S_RUNNER_HTTP_TIMEOUT_SECS (default 240), K8S_RUNNER_HTTP_PROBE_TIMEOUT_SECS (default 30), K8S_RUNNER_HTTP_POLL_INTERVAL_SECS (default 1).

Node Control

The Kubernetes deployer does not wire a node-control handle into managed scenario deployments. A scenario built with with_node_control() compiles against this backend, but runtime restart calls fail. For node lifecycle control on Kubernetes, use the Kubernetes ManualCluster below.

Manual Mode

K8sDeployer::manual_cluster_from_descriptors(descriptors) (or ManualCluster::from_topology) installs the same Helm release, discovers every node’s ports, then scales all node Deployments to zero so your code decides when each node starts:

let deployer = OpenRaftKvK8sDeployer::new();
let cluster = deployer
    .manual_cluster_from_descriptors(OpenRaftKvTopology::new(3))
    .await?;

cluster.start_node("node-0").await?;
cluster.start_node("node-1").await?;
cluster.wait_network_ready().await?;
cluster.restart_node("node-0").await?;
cluster.stop_all();

Start, stop, and restart are implemented by patching Deployment replicas between 0 and 1 and waiting for the rollout. start_node_with accepts StartNodeOptions, with two k8s-specific limits: persist_dir / snapshot_dir are rejected, and peer selection or config overrides require the environment to implement cfgsync override artifacts (cfgsync_service + build_cfgsync_override_artifacts); the override is pushed to the in-cluster cfgsync service through a temporary port-forward before the node starts. The failover demonstration uses this path end to end: cargo run -p openraft-kv-examples --bin openraft_kv_k8s_failover. Contrast with the declarative local variant in ManualCluster: Imperative Node Control.

Attaching to an Existing Cluster

Existing-cluster mode is supported with a k8s descriptor: with_existing_cluster(ExistingCluster::for_k8s_selector("app.kubernetes.io/instance=tf-runner")) (optionally namespaced with for_k8s_selector_in_namespace). Services matching the selector are listed, each service’s single TCP NodePort (or the port named http/api) becomes the node endpoint, and clients are built via Application::external_node_client. deploy_with_metadata returns K8sDeploymentMetadata (namespace + label selector) so a later scenario can attach to the stack this one installed. See Existing and External Clusters.

Observability and Cleanup

Observability inputs resolve exactly as in compose (LOGOS_BLOCKCHAIN_* env vars merged with the scenario capability), and TESTNET_PRINT_ENDPOINTS prints Prometheus/Grafana and per-node pprof endpoints. Cleanup uninstalls the Helm release and deletes the namespace (Kubernetes API first, kubectl delete namespace fallback), after killing any port-forward processes. Set K8S_RUNNER_PRESERVE to keep the release and namespace for inspection.

Requirements recap:

Shared Cluster Provisioning

App composition and uniform scenarios share a cluster-provisioning model. A request describes the cluster source and required behavior. A provisioner returns a backend-independent ClusterHandle<E> and registers any managed lifetime with the app cleanup stack.

One Request, Three Sources

ClusterRequest<E> separates what the test needs from how a backend supplies it:

let managed = ClusterRequest::<QueueEnv>::managed(QueueTopology::new(3));
let attached = ClusterRequest::<QueueEnv>::attached(existing_cluster);
let external = ClusterRequest::<QueueEnv>::external(node_sources);

Managed and attached sources can also include external nodes with with_external_nodes(...). This is useful when one logical cluster combines framework-visible nodes from more than one source.

Requesting Behavior

The request carries requirements that are meaningful across backends:

Backends may support different combinations of these requirements. The Capability Matrix records current coverage.

Provisioning Inside an AppDeployment

DeployContext is parameterized by a ClusterProvisioner. Its deploy_cluster method is the app-layer entry point:

#[async_trait]
impl AppDeployment<AppHostEnv> for QueueApp {
    type Handle = ClusterHandle<QueueEnv>;

    async fn deploy(
        self,
        ctx: &mut DeployContext<AppHostEnv>,
    ) -> Result<Self::Handle, DynError> {
        ctx.deploy_cluster(ClusterRequest::<QueueEnv>::managed(self.topology))
            .await
    }
}

DeployContext::deploy_cluster requests the full common node-control surface because app workloads receive the returned cluster handle directly. The local convenience deploy_local_cluster expresses the common managed, eager case. Use deploy_cluster when ownership mode, start mode, or policy must be visible in the app definition.

with_app(app) selects the default local provisioner. with_app_using(app, provisioner) supplies another provisioner. The root deployment must implement AppDeployment<E, P> for that provisioner type; code written only as AppDeployment<E> uses the default local type.

The Returned Handle

ClusterHandle<E> presents the common runtime surface:

• clients: node_clients, clients, first_client, node_client;
• cluster description: deployment, node_count, control_profile;
• node operations when present: start_node, stop_node, restart_node, wait_node_ready;
• cluster readiness: wait_network_ready;
• network effects when present: network_control.

Unavailable operations return an error or None; callers can inspect control_profile() when behavior depends on ownership mode.

The handle does not own managed lifetime. The provisioner returns a private cleanup guard alongside the runtime surfaces. DeployContext moves that guard into the scenario cleanup stack, which runs in reverse acquisition order on normal completion and partial deployment failure.

Backend Boundary

ClusterProvisioner<E> has one operation:

#[async_trait]
pub trait ClusterProvisioner<E: Application>: Clone + Send + Sync + 'static {
    async fn provision_cluster(
        &self,
        request: ClusterRequest<E>,
    ) -> Result<ClusterUnit<E>, DynError>;
}

A backend implementation translates the request into concrete resources, clients, control adapters, readiness, and cleanup. ClusterUnit<E> carries these values from the provisioner; applications normally use the resulting ClusterHandle<E>.

The local provisioner currently supports managed and external sources. Attached support and equivalent Compose/Kubernetes app provisioners require backend implementations, but not another application-composition model.

Relation to Other Entry Patterns

• A uniform scenario asks its deployer to provision the scenario’s primary cluster.
• A composed stack asks its DeployContext to provision one or more child clusters.
• An attached or external test changes ClusterSource, while workloads keep using clients and available controls.
• ManualCluster uses local provisioning machinery directly and gives imperative code responsibility for sequencing.

The entry patterns differ in who describes and drives the test. They do not need separate definitions of what a cluster is, which controls it exposes, or who tears it down.

See Also

• Existing and External Clusters: declaring non-managed sources in ordinary scenarios.
• AppDeployment and DeployContext: composing cluster and process children.
• Handle Ownership and Teardown: the lifetime boundary in detail.
• Readiness, Retry, and Cleanup: the policies carried by a request.

Existing and External Clusters

Scenarios can run against nodes the framework did not deploy: an attached existing cluster, standalone external endpoints, or a mix.

Every scenario draws its node clients from three source classes: managed nodes the deployer spawns, attached nodes discovered in an existing cluster, and external nodes named by static endpoints. The builder records which sources you want; the deployer resolves them into one NodeClients inventory at deploy time.

The Source Model

The source model uses these types from testing-framework-core:

ExistingCluster is constructed with for_k8s_selector(selector), for_k8s_selector_in_namespace(namespace, selector), for_compose_project(project), or for_compose_services(project, services). ExternalNodeSource::new(label, endpoint) wraps a plain endpoint string.

The mode is derived, not set: a scenario with only a topology is Managed; adding an existing cluster makes it ExistingCluster; with_external_only makes it ExternalOnly. Invalid combinations (managed and attached at once) are unrepresentable. Each mode maps to a ClusterControlProfile, which workloads can consult to know whether the framework owns node lifecycles (framework_owns_lifecycle() is true only for FrameworkManaged).

Builder Methods

use testing_framework_core::scenario::{ExistingCluster, ExternalNodeSource};

// Attach to a running compose project instead of deploying nodes.
let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3))
    .with_existing_cluster(ExistingCluster::for_compose_project("compose-stack-1234".into()))
    .with_workload(KvWriteWorkload::new().operations(100))
    .build()?;

// Add a standalone external endpoint alongside managed nodes.
let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(2))
    .with_external_node(ExternalNodeSource::new(
        "staging-gateway".into(),
        "http://staging.example.net:8080".into(),
    ))
    .build()?;

External nodes compose with every mode: managed + external and attached + external are both valid hybrids.

From Source to Typed Client

External and attached sources become typed clients through one hook on the Application trait:

fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;

The default implementation errors with “external node sources are not supported”; an application opts in by parsing source.endpoint() and constructing its client. The local deployer additionally falls back to a generic parser that resolves http://host:port endpoints and builds the client from the socket address when the app has not overridden the hook.

Resolution at Runtime

At deploy time the scenario’s sources become a SourceOrchestrationPlan, and each deployer supplies a SourceProviders set: a managed provider (the clients it just deployed), an attach provider, and an external provider. orchestrate_sources_with_providers resolves the plan:

flowchart LR
    P[SourceOrchestrationPlan] --> M[managed provider<br/>deployer-spawned clients]
    P --> A[attach provider<br/>discover existing cluster]
    P --> X[external provider<br/>external_node_client]
    M --> N[NodeClients]
    A --> N
    X --> N

The final inventory is ordered managed, then attached, then external. Managed mode with zero managed nodes is rejected; existing-cluster and external-only modes require at least one resolved client overall.

Per-deployer attach support:

• Local: no attach. ProcessDeployer rejects ClusterMode::ExistingCluster outright; external nodes are supported.
• Compose: requires a compose descriptor. Services are taken from the descriptor or discovered from the running project; each container’s labeled API port is inspected and turned into an ExternalNodeSource fed to external_node_client. Attached mode also wires restart/stop node control. See Compose Deployer.
• K8s: requires a k8s descriptor. Services matching the label selector are listed in the namespace (default default); each service’s single TCP NodePort (preferring ports named http or api) becomes the endpoint. See Kubernetes Deployer.

Deploy-Then-Attach

Both container deployers return metadata that converts back into an attach descriptor, so one process can deploy a stack and a second scenario can attach to it:

let (runner, metadata) = ComposeDeployer::<KvEnv>::new()
    .deploy_with_metadata(&scenario)
    .await?;

// Later, or elsewhere: attach to the same project.
let attached = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3))
    .with_existing_cluster_from(&metadata)?
    .build()?;

K8sDeployer::deploy_with_metadata provides the equivalent K8sDeploymentMetadata (namespace + label selector).

Use Cases

• Staging and live networks. Point with_external_only_nodes at long-lived endpoints and run workloads and expectations against them; the framework never touches their lifecycle (ExternalUncontrolled).
• Shared test stacks. Deploy a compose or k8s stack once, attach many fast scenarios to it, and preserve the stack between runs with the deployer preserve env vars (see Readiness, Retry, and Artifact Preservation).
• Hybrid scenarios. Combine managed nodes with an external dependency, for example a locally deployed cluster that must interoperate with a fixed remote peer.

Manual clusters have their own external hooks: add_external_sources and add_external_clients on ManualCluster merge external endpoints into an imperatively driven cluster (see ManualCluster).

Binary Providers

Binary providers resolve the executable a local node process runs. The source can be a path, an env var, a build command, a download, or an ordered fallback chain.

Every local node launch has exactly one provider selected on its LocalProcessSpec. Providers live in testing_framework_runner_local::binary and are re-exported from the crate root. They apply to the Local Deployer only; compose and k8s nodes run container images instead (see the Capability Matrix).

The Trait

pub trait BinaryProvider: Send + Sync {
    fn try_resolve(&self) -> Result<Option<PathBuf>, BinaryProviderError>;
    fn display(&self) -> String;
    fn cache_key(&self) -> String;

    // Provided: cache lookup, then resolve_uncached.
    fn resolve(&self) -> Result<PathBuf, BinaryProviderError> { /* ... */ }
    fn resolve_uncached(&self) -> Result<PathBuf, BinaryProviderError> { /* ... */ }
}

pub type BinaryProviderRef = Arc<dyn BinaryProvider>;

try_resolve returns Ok(None) when the provider is valid but cannot produce a binary in the current environment, which is not an error. Standalone, resolve turns None into BinaryProviderError::NotFound; inside a FallbackBinaryProvider, None means “try the next provider”. Other errors (a failed build, a checksum mismatch) abort resolution immediately.

The Providers

PathBinaryProvider::new(path): a deterministic explicit path. No filesystem search, no PATH lookup.

EnvBinaryProvider::new("MY_NODE_BIN"): the standard override hook. LocalProcessSpec::new(env_var) installs one of these by default.

BuildBinaryProvider delegates to any command:

BuildBinaryProvider {
    command: BuildCommand::new("cargo").with_args(["build", "-p", "kvstore-node"]),
    output_path: "target/debug/kvstore-node".into(), // relative to working_dir
    working_dir: Some(workspace_root),                // default: current dir
    lock_dir: None,                                   // default: <working_dir>/.tf-binaries
}

The command is not Cargo-specific; it can invoke a Make target, shell script, or cache fetch. After the command succeeds, the configured output_path must exist or resolution fails with MissingBuildOutput.

DownloadBinaryProvider fetches into a cache directory (default target/.tf-binaries under the current directory):

• DownloadUrl::Fixed(url) or DownloadUrl::Env(var) selects the source.
• DownloadChecksum::Fixed(sha256) or DownloadChecksum::Env(var) enables SHA-256 verification; mismatches fail with ChecksumMismatch before anything is written to the final path.
• A DownloadProcessor post-processes artifacts that are not directly executable (archives, bundles). It receives the verified download and must materialize the executable at the output path. DownloadProcessorFn::new(cache_key, closure) (or .with_processor_fn(...)) is the lightweight adapter; the cache_key is part of cache identity, so changing your extraction logic invalidates previously prepared binaries.
• On Unix, the result is marked executable (0o755). Downloads are staged through temporary .download/.part files and renamed into place.

FallbackBinaryProvider::new([a, b, ...]): an ordered chain, tried first to last. From the launch spec’s perspective it is still a single provider.

Caching and Cache Identity

Successful resolutions are cached per process in a global map keyed by cache_key(), so repeated node starts with the same provider config do not rebuild, redownload, or re-scan. Cache identity encodes the full request:

• path:<path>, env:<var>
• build:<command>:<output_path>:<working_dir>
• download:<url-or-env>:<checksum-or-env>:<processor-key>:<cache_dir>
• fallback: the members’ keys joined with commas

Change any component and you get a fresh resolution. The download provider also caches on disk: the cached file name hashes the URL, resolved checksum, and processor key, so an already-downloaded binary is reused across processes without refetching.

Concurrent Resolution Locking

Builds and downloads may be triggered by several test processes at once (e.g. cargo nextest running integration tests in parallel). Providers that materialize files take a cross-process file lock before doing work: a lock file created with create_new under .tf-binaries (build) or the download cache dir, retried every 200 ms for up to 10 minutes, then BinaryProviderError::LockTimeout. The lock file is removed when the guard drops.

A killed test process can leave a stale lock file behind. If resolution hangs and then times out, look for leftover *.lock files under .tf-binaries and delete them.

Worked Example: kvstore’s Fallback Chain

From examples/kvstore/testing/integration/src/local_env.rs, which prefers an explicit env override and otherwise builds from source:

use std::{path::PathBuf, sync::Arc};
use testing_framework_runner_local::{
    BinaryProviderRef, BuildBinaryProvider, BuildCommand, EnvBinaryProvider,
    FallbackBinaryProvider, LocalProcessSpec,
};

fn kvstore_binary_provider() -> FallbackBinaryProvider {
    let providers: [BinaryProviderRef; 2] = [
        Arc::new(EnvBinaryProvider::new("KVSTORE_NODE_BIN")),
        Arc::new(BuildBinaryProvider {
            command: BuildCommand::new("cargo")
                .with_args(["build", "-p", "kvstore-node", "--bin", "kvstore-node"]),
            output_path: PathBuf::from(format!(
                "target/debug/kvstore-node{}",
                std::env::consts::EXE_SUFFIX
            )),
            working_dir: Some(workspace_root()),
            lock_dir: None,
        }),
    ];
    FallbackBinaryProvider::new(providers)
}

fn local_process_spec() -> LocalProcessSpec {
    LocalProcessSpec::new("KVSTORE_NODE_BIN")
        .with_binary_provider(kvstore_binary_provider())
        .with_rust_log("kvstore_node=info")
}

First run: KVSTORE_NODE_BIN is unset, the env provider yields None, the build provider compiles the node under the workspace lock, and the path is cached for the rest of the process. Set KVSTORE_NODE_BIN=/path/to/kvstore-node to skip the build entirely, which is useful for prebuilt release binaries or mixed-version clusters (via local_process_spec_for_node, see Local Deployer).

Readiness, Retry, and Artifact Preservation

DeploymentPolicy is the single policy struct that controls readiness gating, deploy retries, and artifact retention across all deployers.

The Policy

From testing-framework-core (core/src/scenario/deployment_policy.rs):

pub struct DeploymentPolicy {
    pub readiness_enabled: bool,
    pub readiness_requirement: HttpReadinessRequirement,
    pub retry_policy: Option<RetryPolicy>,
    pub cleanup_policy: CleanupPolicy,
}

pub struct RetryPolicy {
    pub max_attempts: usize,
    pub base_delay: Duration,
    pub max_delay: Duration,
}

pub struct CleanupPolicy {
    pub preserve_artifacts: bool,
}

Defaults: readiness_enabled: true, readiness_requirement: HttpReadinessRequirement::AllNodesReady, retry_policy: None, preserve_artifacts: false. HttpReadinessRequirement is AllNodesReady, AnyNodeReady, or AtLeast(usize).

Set it on the builder:

use std::time::Duration;
use testing_framework_core::scenario::{
    CleanupPolicy, DeploymentPolicy, HttpReadinessRequirement, RetryPolicy,
};

let scenario = KvScenarioBuilder::deployment_with(|_| KvTopology::new(3))
    .with_deployment_policy(DeploymentPolicy {
        readiness_enabled: true,
        readiness_requirement: HttpReadinessRequirement::AtLeast(2),
        retry_policy: Some(RetryPolicy::new(
            5,
            Duration::from_millis(500),
            Duration::from_secs(5),
        )),
        cleanup_policy: CleanupPolicy::new(true),
    })
    .build()?;

To adjust only the requirement, with_http_readiness_requirement(...) is the shortcut.

Readiness

readiness_enabled and readiness_requirement gate the post-spawn probe pass in every deployer. Each backend also has its own deployer-level switch that must agree (ProcessDeployer::with_membership_check(bool), ComposeDeployer::with_readiness(bool), K8sDeployer::with_readiness(bool)), so effective readiness is deployer switch && policy.readiness_enabled. The probe shape (HTTP path vs TCP) comes from the application environment; see the per-deployer chapters (Local, Compose, K8s).

Retry

retry_policy drives the local deployer’s spawn-and-readiness loop: on failure, all nodes from the attempt are dropped and the cluster is respawned with exponential backoff (from base_delay, capped at max_delay, with jitter) up to max_attempts. When retry_policy is None, the local deployer falls back to its built-in default of 3 attempts, 250 ms base delay, 2 s max delay.

The Compose and Kubernetes deployers currently honor the readiness fields of the policy but do not repeat deployment on failure; retry_policy has no effect on those backends today.

Artifact Preservation

cleanup_policy.preserve_artifacts controls artifact and tempdir retention, not teardown ordering. Teardown itself always follows the runner’s cleanup-guard chain (see Handle Ownership and Teardown); this flag only decides whether per-node working directories survive it.

The local orchestrator computes retention as:

policy.cleanup_policy.preserve_artifacts || keep_tempdir_from_env() // TF_KEEP_LOGS

so either the policy flag or TF_KEEP_LOGS=1 (also true/yes) keeps every node’s working directory (configs, on-disk state, anything the process wrote) after the run. Panicking tests preserve working directories regardless.

The container deployers preserve through env vars rather than the policy: COMPOSE_RUNNER_PRESERVE / TESTNET_RUNNER_PRESERVE keep the compose stack and workspace, K8S_RUNNER_PRESERVE keeps the Helm release and namespace. See Diagnostics and Retained Artifacts.

Part VI — Extending and Reference

This part documents the extension points, the crate map, and the boundary rules that keep the framework application-agnostic.

• Public Extension Points — the traits you implement to plug in
• Crate and API Map — which concept lives in which crate
• Framework vs Application Boundaries — what belongs where, and how it is enforced

Public Extension Points

This chapter lists every trait you implement to plug your application into the framework.

The framework never imports your application. It defines public traits that your integration crate implements and calls them at defined points in the run lifecycle. Each entry below links to the corresponding chapter.

Environment and Topology

Application is the root of every integration. It bundles the backend-specific types the scenario engine is generic over: a deployment descriptor, a node client, and a node config. The three methods have working defaults: override external_node_client to support external sources, build_node_client to support deployer-discovered nodes, and node_readiness_path when your health endpoint is not /.

#[async_trait]
pub trait Application: Send + Sync + 'static {
    type Deployment: DeploymentDescriptor + Clone + 'static;
    type NodeClient: Clone + Send + Sync + 'static;
    type NodeConfig: Clone + Send + Sync + 'static;

    fn external_node_client(source: &ExternalNodeSource) -> Result<Self::NodeClient, DynError>;
    fn build_node_client(access: &NodeAccess) -> Result<Self::NodeClient, DynError>;
    fn node_readiness_path() -> &'static str; // default "/"
}

Plugs in as the E type parameter of ScenarioBuilder<E>, Workload<E>, Expectation<E>, and every deployer.

DeploymentProvider<D> builds the deployment descriptor a scenario runs against, optionally driven by a DeploymentSeed for reproducible generation. ScenarioBuilder::new accepts one; ScenarioBuilder::with_deployment wraps a fixed value in the built-in FixedDeploymentProvider.

pub trait DeploymentProvider<D: DeploymentDescriptor>: Send + Sync {
    fn build(&self, seed: Option<&DeploymentSeed>) -> Result<D, DynTopologyError>;
}

Scenario Behavior

Workload<E> describes an action sequence executed during the run. start receives the RunContext<E> (node clients, extensions, run metrics) and runs concurrently with other workloads. A workload can bundle its own checks via expectations().

#[async_trait]
pub trait Workload<E: Application>: Send + Sync {
    fn name(&self) -> &str;
    fn expectations(&self) -> Vec<Box<dyn Expectation<E>>> { Vec::new() }
    fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) }
    async fn start(&self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

Registered with with_workload / with_workload_boxed on the builder.

Expectation<E> defines a check evaluated during or after the run. start_capture records a baseline, check_during_capture is the optional fail-fast hook polled during the run, and evaluate delivers the verdict at the end.

#[async_trait]
pub trait Expectation<E: Application>: Send + Sync {
    fn name(&self) -> &str;
    fn init(&mut self, descriptors: &E::Deployment, metrics: &RunMetrics) -> Result<(), DynError> { Ok(()) }
    async fn start_capture(&mut self, ctx: &RunContext<E>) -> Result<(), DynError> { Ok(()) }
    async fn check_during_capture(&mut self, ctx: &RunContext<E>) -> Result<(), DynError> { Ok(()) }
    async fn evaluate(&mut self, ctx: &RunContext<E>) -> Result<(), DynError>;
}

Registered with with_expectation / with_expectation_boxed.

RuntimeExtensionFactory<E> prepares one typed value after deployment (node clients are available) and before workloads start. The value is stored by TypeId and retrieved in workloads via ctx.extension::<T>() / ctx.require_extension::<T>(). Return PreparedRuntimeExtension::new(value), ::with_cleanup(value, guard), or ::from_task(value, join_handle) to tie a background task’s lifetime to the run.

#[async_trait]
pub trait RuntimeExtensionFactory<E: Application>: Send + Sync {
    async fn prepare(
        &self,
        deployment: &E::Deployment,
        node_clients: NodeClients<E>,
    ) -> Result<PreparedRuntimeExtension, DynError>;
}

Registered with with_runtime_extension_factory. Registering two factories that produce the same extension type fails at prepare time with duplicate runtime extension type registered.

Observation

Observer owns the app-side logic of the continuous observation runtime: init builds retained state from the source set, poll advances it each cycle and emits delta events, snapshot renders the current view. The runtime handles scheduling, history, and error tracking.

#[async_trait]
pub trait Observer: Send + Sync + 'static {
    type Source: Clone + Send + Sync + 'static;
    type State: Send + Sync + 'static;
    type Snapshot: Clone + Send + Sync + 'static;
    type Event: Clone + Send + Sync + 'static;

    async fn init(&self, sources: &[ObservedSource<Self::Source>]) -> Result<Self::State, DynError>;
    async fn poll(&self, sources: &[ObservedSource<Self::Source>], state: &mut Self::State)
        -> Result<Vec<Self::Event>, DynError>;
    fn snapshot(&self, state: &Self::State) -> Self::Snapshot;
}

SourceProvider<S> returns the current source set before each cycle, which lets the observed population change mid-run. Use StaticSourceProvider for a fixed set.

#[async_trait]
pub trait SourceProvider<S>: Send + Sync + 'static {
    async fn sources(&self) -> Result<Vec<ObservedSource<S>>, DynError>;
}

SourceProviderFactory<E, S> builds the provider once node clients exist. Any Fn(&E::Deployment, NodeClients<E>) -> Result<BoxedSourceProvider<S>, DynError> closure implements it. All three plug into a scenario through ObservationExtensionFactory<E, O>, which is itself a RuntimeExtensionFactory; see examples/openraft_kv/testing/integration/src/observation.rs for a complete implementation.

Application Composition

AppDeployment<E, P> prepares one reusable application preset, such as a process, child cluster, or composed stack, and returns a typed access or control handle. Framework adapters register managed resource lifetime separately with scenario cleanup. AppHandle is blanket-implemented for any Clone + Send + Sync + 'static type.

#[async_trait]
pub trait AppDeployment<E, P = LocalClusterProvisioner>: Send + 'static
where
    E: Application,
{
    type Handle: AppHandle;
    async fn deploy(self, ctx: &mut DeployContext<E, P>) -> Result<Self::Handle, DynError>;
}

Registered with AppScenarioBuilderExt::with_app, which wraps it in an AppDeploymentFactory (a RuntimeExtensionFactory). Compose children inside deploy via ctx.deploy(...) / ctx.deploy_and_expose(...). See Handle Ownership and Teardown for handle access and cleanup semantics.

Deployment Backends

Deployer<E, Caps> is the contract every backend implements: turn a built Scenario into a Runner<E>. ProcessDeployer (local), ComposeDeployer, and K8sDeployer are the in-repo implementations; Caps carries capability markers such as NodeControlCapability.

#[async_trait]
pub trait Deployer<E: Application, Caps = ()>: Send + Sync {
    type Error;
    async fn deploy(&self, scenario: &Scenario<E, Caps>) -> Result<Runner<E>, Self::Error>;
}

NodeControlHandle<E> is the deployer-agnostic control surface behind node-control scenarios: start_node(_with), stop_node, restart_node(_with), wait_node_ready, node_client, and node_pid. Every method has a default that returns a “not supported by this deployer” error, so backends implement only what they support. ClusterWaitHandle<E> provides the cluster-wide wait_network_ready operation. Both are combined by ManualClusterHandle<E> in core::runtime::manual, the interface behind ManualCluster.

ObservabilityCapabilityProvider lets deployers read telemetry endpoints out of whatever capability marker a scenario was built with; it is implemented for (), NodeControlCapability, and ObservabilityCapability. You only implement it when defining a new capability marker type.

Local Binary Resolution

BinaryProvider resolves the executable path for a locally spawned node process. Implementations return Ok(None) when valid but unable to resolve, which is how FallbackBinaryProvider chains providers. The default resolve caches per process by cache_key.

pub trait BinaryProvider: Send + Sync {
    fn try_resolve(&self) -> Result<Option<PathBuf>, BinaryProviderError>;
    fn display(&self) -> String;
    fn cache_key(&self) -> String;
    // provided: resolve(), resolve_uncached()
}

Built-in implementations: PathBinaryProvider, EnvBinaryProvider, BuildBinaryProvider, DownloadBinaryProvider, FallbackBinaryProvider. DownloadProcessor post-processes a checksum-verified download (for example, unpacking an archive) into the executable; DownloadProcessorFn adapts a closure with a stable cache_key so changed preparation logic invalidates the cache.

pub trait DownloadProcessor: Send + Sync {
    fn process(&self, artifact: &Path, output: &Path) -> Result<(), DownloadProcessorError>;
    fn cache_key(&self) -> &str;
}

Attaching Sources

IntoExistingCluster converts a value into the typed ExistingCluster descriptor accepted by with_existing_cluster_from. It is implemented for ExistingCluster and &ExistingCluster; implement it for your own environment-selection types to keep attach logic in one place. External endpoints use ExternalNodeSource values directly and pair with Application::external_node_client.

The required extension points depend on the entry pattern: a uniform managed cluster needs Application and the scenario traits, an AppHost stack adds AppDeployment, and attached clusters add the source traits. See Choosing an Entry Pattern. For where each implementation should live, see Framework vs Application Boundaries and the crate-level view in Crate and API Map.

Crate and API Map

This chapter maps which crate owns which concept, what each one exports, and how they depend on each other.

The workspace splits into three layers: the app-agnostic core, the deployment backends, and the cfgsync configuration pipeline. Example applications live in their own workspace layout under examples/ and depend on the framework, never the other way around.

graph BT
    art[cfgsync-artifacts]
    cc[cfgsync-core] --> art
    ca[cfgsync-adapter] --> cc
    ca --> art
    cr[cfgsync-runtime] --> ca
    core[testing-framework-core] --> ca
    local[testing-framework-runner-local] --> core
    compose[testing-framework-runner-compose] --> core
    k8s[testing-framework-runner-k8s] --> core
    k8s --> cc
    k8s --> art
    app[testing-framework-app] --> core
    app --> local

testing-framework-core

Path: testing-framework/core. The scenario engine and everything app-agnostic: builder, runtime, topology, observation, sources, capabilities. Every other framework crate depends on it.

Key builder entry points: ScenarioBuilder::with_deployment, ::new(provider), and the capability-gated variants with_node_control() and with_observability(). ObservabilityBuilderExt and CoreBuilderExt live here too.

testing-framework-app

Path: testing-framework/app. The app layer for heterogeneous stacks: singleton processes, extra clusters, or several applications composed into one system. Depends on core plus the local deployer; the app layer is local-only today (see Backend Scope).

Deployment Backends

Each backend implements Deployer<E> for its environment trait and returns the same core Runner<E>.

testing-framework-runner-local (testing-framework/deployers/local) spawns nodes as local processes. Exports ProcessDeployer, ManualCluster, NodeManager, the LocalDeployerEnv / LocalBinaryApp environment traits with config/port helpers (LocalProcessSpec, LocalNodePorts, build_local_cluster_node_config, …), process primitives (LaunchSpec, NodeEndpoints, ProcessNode), and the whole binary module (BinaryProvider and its implementations). Honors TF_KEEP_LOGS for tempdir retention.

testing-framework-runner-compose (.../compose) renders a Docker Compose stack. Exports ComposeDeployer, ComposeDeployEnv, descriptor builders (ComposeDescriptor, NodeDescriptor), compose lifecycle commands (compose_up, compose_down, dump_compose_logs), and the Docker config-server support used to serve cfgsync artifacts to containers.

testing-framework-runner-k8s (.../k8s) installs a Helm release. Exports K8sDeployer, K8sDeployEnv, ManualCluster (K8s variant), Helm/chart-value infrastructure (HelmInstallSpec, RunnerChartValues, render_binary_config_node_chart_assets, …), and wait/cleanup helpers. Depends directly on cfgsync-core and cfgsync-artifacts for artifact delivery.

cfgsync

cfgsync is the typed pipeline that turns app config into per-node files: app config → registration snapshot → per-node artifact sets → backend rendering. Consumed by the compose and k8s deployers (locally, configs are written straight to disk). See Static Artifacts and cfgsync.

Examples Workspace Layout

Every example app follows the same four-part shape under examples/<app>/:

examples/kvstore/
├── kvstore-node/            # the application binary under test
├── testing/
│   ├── integration/         # crate kvstore-runtime-ext: Application impl,
│   │                        #   local/compose/k8s env impls, observation
│   └── workloads/           # crate kvstore-runtime-workloads: Workloads + Expectations
└── examples/                # crate kvstore-examples: runnable bins

The naming is uniform: <app>-runtime-ext, <app>-runtime-workloads, <app>-examples. nats and redis_streams have no node crate because they run upstream binaries or images. multi_app uses an acceptance-suite layout instead: a job-worker/ binary crate, a fixture/ crate (multi-app-fixture: the stack deployment, handles, workload, and expectation), and an e2e/ crate (multi-app-e2e) whose integration tests drive the fixture. It demonstrates application composition.

Run any example bin with:

cargo run -p kvstore-examples --bin kvstore_basic_convergence

Note: the dependency arrows only ever point from examples toward the framework and from backends toward core. If you find yourself wanting an arrow in the other direction, read Framework vs Application Boundaries. The trait-level view of the same surface is in Public Extension Points.

Framework vs Application Boundaries

This chapter covers the rules that keep the framework application-agnostic and how they are enforced in practice.

Ownership and Design Boundaries explains the ownership split. This chapter covers the concrete rules, enforcement mechanisms, and signs that code is in the wrong layer.

The Rule

Dependencies point in exactly one direction: application repositories depend on framework crates, never the reverse. The framework knows applications only through the traits in Public Extension Points; everything app-specific (node configs, HTTP clients, readiness semantics, observers, workloads) lives in the application’s own integration crates.

The in-repo examples/ workspace models the application side: each app has an integration crate (<app>-runtime-ext) implementing Application and the per-backend environment traits, and a workloads crate (<app>-runtime-workloads) implementing Workload and Expectation. No framework crate names an example app in its Cargo.toml; see the dependency diagram in Crate and API Map.

graph LR
    subgraph Application side
        ext["&lt;app&gt;-runtime-ext<br/>Application, env impls, Observer"]
        wl["&lt;app&gt;-runtime-workloads<br/>Workload, Expectation"]
    end
    subgraph Framework side
        core[testing-framework-core]
        appl[testing-framework-app]
        runners[deployers]
    end
    ext --> core
    ext --> runners
    wl --> core
    appl --> core
    runners --> core

Traits cross the boundary; concrete application types never do.

What belongs where:

Enforcement Mechanisms

The type system. The scenario engine is generic over Application, so core code physically cannot reference your node client or config, because there is no concrete type to name. The app layer goes further: AppHostEnv sets NodeClient = () and its build_node_client returns an error, forcing application clients to travel as typed handles owned by the app side rather than leaking into the environment.

Runtime registration errors. Two rules are enforced with hard errors instead of silent replacement:

• Registering two runtime extension factories that produce the same type fails at prepare time with duplicate runtime extension type registered: .... This is also why a scenario takes exactly one with_app; compose multiple apps inside one root AppDeployment instead.
• Exposing a handle twice under the same type and name fails with app handle is already exposed: ... (AppDeployError::DuplicateHandle).

The boundary check script. scripts/run/check-boundaries.sh guards the adopter side of the line. What it actually does:

• resolves a sibling adopter checkout at ../nomos-node/tests/testing_framework/lb-topology and fails if it is missing;
• greps that crate’s src/ and Cargo.toml for extension-specific identifiers (cfgsync, ComposeDeployEnv, K8sDeployEnv, runner-compose, runner-k8s, DEFAULT_CFGSYNC_PORT, DEFAULT_ASSETS_STACK_DIR) and fails on any hit.

The same rule can be applied to other integration crates: a topology-level crate remains backend-agnostic, so references to a specific deployer or cfgsync internals are treated as violations. Backend names belong in the per-backend environment modules; compare local_env.rs, compose_env.rs, and k8s_env.rs in examples/openraft_kv/testing/integration/src/.

Crate docs as contract. testing-framework-app states the ownership boundary in its crate docs: implement AppDeployment in the application repository, compose children through DeployContext, and let handles own deployed resources. The multi_app README says the same from the other direction: for composed systems, prefer the app-layer shape “instead of building a fake outer cluster or adding app-specific code to TF”.

Signs Your Code Is on the Wrong Side

Symptoms that application code has leaked into the framework:

• A config template, launch flag, or port convention for one specific application sitting in testing-framework/ or cfgsync/.
• A framework crate importing an example (or adopter) crate, or matching on an application name.
• A “generic” helper in core whose only caller is one app and whose parameters mirror that app’s config fields.

Symptoms that framework mechanics are being re-implemented in the application repo:

• Hand-rolled process spawn/kill/teardown code where LocalProcessApp or LocalAppCluster would do.
• A custom polling loop with history and error tracking that duplicates the observation runtime; implement Observer instead.
• Re-implementing binary resolution, caching, or fallback chains instead of configuring BinaryProvider types.
• A bespoke “wait until cluster healthy” loop instead of readiness closures plus DeploymentPolicy (see Readiness, Retry, and Artifact Preservation).

A framework addition should compile and make sense with a different application plugged in. Application-specific code belongs in the application repository.

Backend Scope of the App Layer

The composition layer is local-only today, and this is visible in the dependency graph: testing-framework-app depends on core and testing-framework-runner-local only, AppHostLocalDeployer is an alias for ProcessDeployer<AppHostEnv>, and DeployContext::deploy_local_cluster / LocalAppCluster require LocalDeployerEnv. The compose and k8s deployers remain single-application. Do not work around this by teaching the framework about your app’s containers: run composed stacks locally, and use the Compose or Kubernetes deployer for uniform clusters. Details in Backend Scope.

Part VII — Operations

These chapters cover running, integrating, and debugging the framework day to day.

• Running the Examples — every runnable binary and what it exercises
• Continuous Integration — how this repository tests itself, and patterns for yours
• Diagnostics and Retained Artifacts — logs, working directories, post-mortems
• Environment Variables — the complete, audited reference
• Troubleshooting — common failures and their causes
• Glossary — terms used throughout the book

Running the Examples

This chapter lists every runnable example binary, the exact command to launch it, and what it needs from your machine.

Conventions

All examples are ordinary binaries run with:

cargo run -p <package> --bin <bin>

Naming encodes the backend: *_basic_* and *_app_host_* run as local processes, *_compose_* need a running Docker daemon, and *_k8s_* need a reachable Kubernetes cluster context (the k8s deployer drives Helm and the cluster API). Compose binaries exit gracefully with a warning when Docker is unavailable, and the k8s binaries skip when the cluster cannot be reached (K8sRunnerError::ClientInit).

Logging uses tracing_subscriber with an env filter; set RUST_LOG to adjust verbosity.

Summary

Binary Resolution for Local Runs

Local examples resolve their node binary through a Binary Provider:

• kvstore and openraft_kv use a FallbackBinaryProvider: an explicit KVSTORE_NODE_BIN / OPENRAFT_KV_NODE_BIN override wins, otherwise a BuildBinaryProvider runs cargo build -p <node-crate> for you. No setup needed.
• queue, pubsub, and metrics_counter use a plain EnvBinaryProvider: you must build the node and point the env var at it: