Cluster Control and Observability

Node control and telemetry endpoints are per-cluster request data. Each ClusterRequest states what the test needs from that cluster; the backend provisioner wires the matching runtime surfaces into the returned ClusterHandle. Operations a cluster does not have fail with a clear error at the call site instead of somewhere inside a workload.

Control as Request Data

ClusterControlRequest (testing-framework/core/src/scenario/provisioning.rs) has two values:

A raw ClusterRequest defaults to None. ClusterApp defaults to Full, because a cluster deployed as the system under test usually wants restarts available; opt out with .with_control(ClusterControlRequest::None):

let scenario = AppHost::scenario()
    .with_app(
        ClusterApp::<QueueEnv>::new(QueueTopology::new(3))
            .with_control(ClusterControlRequest::None),   // clients only
    )
    .build()?;

What Full buys depends on the backend: local grants the complete surface including StartNodeOptions, compose grants restart (plus stop for attached clusters), and k8s grants start/stop/restart by scaling per-node deployments. The Backend Capability Matrix records the details.

Every handle also reports a ClusterControlProfile describing who owns the lifecycle: FrameworkManaged (provisioned eagerly, torn down by the framework), ManualControlled (provisioned on demand, your code starts nodes), ExistingClusterAttached, or ExternalUncontrolled. framework_owns_lifecycle() is true only for FrameworkManaged.

Node Control Without ManualCluster

Restarting nodes from a declarative workload does not require ManualCluster. The handle carries the control surface:

async fn start(&self, ctx: &RunContext<AppHostEnv>) -> Result<(), DynError> {
    let cluster = ctx.require_app::<ClusterHandle<QueueEnv>>()?;

    cluster.restart_node("node-1").await?;
    cluster.wait_node_ready("node-1").await?;
    Ok(())
}

When control was not requested (or the backend cannot provide an operation), the call returns an error such as "cluster node control is not available" — explicit partial support at run time. Chaos and Controlled Failure shows full restart scenarios built this way, including the reusable ClusterRestartChaos workload. ManualCluster remains the imperative API for tests that control the entire node lifecycle themselves; see ManualCluster: Imperative Node Control.

The Handle’s Control Surface

ClusterHandle<E> forwards these operations to the backend’s NodeControlHandle (testing-framework/core/src/scenario/control.rs):

StartedNode<E> is a plain pair: the node name and a fresh E::NodeClient.

wait_network_ready() is the matching cluster-wide wait, backed by the provisioner’s ClusterWaitHandle. Inside workloads, prefer waiting on observed application state instead.

Start Modes

ClusterStartMode selects when managed nodes start:

• Eager (default): the provisioner starts and readiness-gates every node before the handle is returned.
• OnDemand: the cluster is prepared but no node runs; the handle’s profile is ManualControlled and your code calls start_node explicitly. Supported by the local and k8s provisioners; the compose provisioner rejects it.

ClusterApp::<QueueEnv>::new(QueueTopology::new(3))
    .with_start_mode(ClusterStartMode::OnDemand)

StartNodeOptions

StartNodeOptions<E> customizes a dynamic start or restart. Overview (full treatment in Part IV: Ports, Peers, Node Config, and Readiness and Persistence, Snapshots, and Recovery Testing):

The local backend honors all of them; k8s rejects persist/snapshot dirs and routes config overrides through cfgsync where the environment supports it (Kubernetes Backend).

Observability as Request Data

ObservabilityInputs (testing-framework/core/src/scenario/observability.rs) carries optional telemetry endpoints:

Set it per cluster:

let observability = ObservabilityInputs {
    metrics_query_url: Some(Url::parse("http://127.0.0.1:19091")?),
    ..ObservabilityInputs::default()
};

let app = ClusterApp::<MetricsCounterEnv>::new(MetricsCounterTopology::new(3))
    .with_observability(observability);

The compose and k8s provisioners resolve the effective inputs by reading the LOGOS_BLOCKCHAIN_METRICS_QUERY_URL, LOGOS_BLOCKCHAIN_METRICS_OTLP_INGEST_URL, and LOGOS_BLOCKCHAIN_GRAFANA_URL environment variables and applying the request’s values as overrides (request wins). The OTLP ingest URL flows into node config preparation; the resolved inputs are stored on the handle. The local provisioner does not populate observability, so local handles report empty metrics.

Expectations read metrics through the handle:

let cluster = ctx.require_app::<ClusterHandle<MetricsCounterEnv>>()?;
let metrics = cluster.metrics()?;             // Prometheus-backed when configured
let total = metrics.counter_value("sum(metrics_counter_increments_total)")?;

metrics() builds a Prometheus-backed Metrics handle from metrics_query_url, or an empty handle when none is configured (is_configured() distinguishes the two). Details, including what telemetry is and is not, are in Telemetry and External Observability.

See Also

• Chaos and Controlled Failure — node control from workloads
• ManualCluster: Imperative Node Control — the imperative alternative
• Telemetry and External Observability — observability inputs in use
• Backend Capability Matrix — backend support by feature