Pipeline replica sets
A Pipeline Replica Set (PRS) runs multiple independent jobs for one HTTP-source pipeline behind a single load-balanced endpoint. It provides one logical pipeline name, a declared replica count, and a stable address for HTTP producers.
Use a PRS when an HTTP ingestion endpoint needs more availability or request capacity than one pipeline job can provide.
Why HTTP sources use replicas
Partitioned sources such as Kafka, Kinesis, and object storage divide their input among parallel subtasks. An HTTP-push source is different: data arrives at a network endpoint, so increasing operator parallelism does not create additional independently reachable endpoints.
For an HTTP-source pipeline:
use replicas to add independently running HTTP endpoints behind one load balancer;
use parallelism for downstream processing where the compiled plan supports it; and
use a normal pipeline, rather than a PRS, for sources that already partition their input.
A PRS addresses three operational requirements:
Ingestion capacity - requests can be distributed across multiple healthy replicas.
Endpoint availability - one replica can restart or recover while other replicas continue accepting requests.
Stable addressing - producers send requests to one endpoint without tracking individual jobs.
Architecture and failure model
Each replica is a complete pipeline job with its own runtime state, checkpoints, and recovery lifecycle. Replicas share the compiled pipeline definition and the load-balanced HTTP endpoint; they do not share in-memory operator state.
Unit of execution
One job
Multiple independent jobs
Scale control
Operator parallelism
Replica count, plus supported downstream parallelism
HTTP endpoint
One job endpoint
One endpoint load-balanced across healthy replicas
Failure impact
Endpoint is unavailable while the job recovers
Other healthy replicas continue serving requests
Checkpoints and state
One recovery history
Independent recovery history per replica
In Kubernetes, one dedicated Service routes requests to the ready replicas in the set. Readiness controls prevent traffic from being sent to a replica that is starting, stopping, or recovering.
Because replicas checkpoint independently, a failed replica recovers from its own latest successful checkpoint without rolling back healthy replicas. Producers should still use normal retry and idempotency practices because an individual request can fail while a replica is being removed from service.
REST API
Authorized API clients can create and manage replica sets directly.
POST /v1/pipeline_replica_sets
Create a PRS, compile its query, and schedule the requested replicas.
GET /v1/pipeline_replica_sets
List replica sets.
GET /v1/pipeline_replica_sets/{id}
Get one PRS, its phase, and its active replica jobs.
PATCH /v1/pipeline_replica_sets/{id}
Change replicas, parallelism, checkpoint interval, or stop state.
DELETE /v1/pipeline_replica_sets/{id}
Delete a PRS after all of its jobs are terminal.
POST /v1/pipeline_replica_sets/{id}/restart
Rotate healthy replicas and revive failed replicas.
POST /v1/pipeline_replica_sets/{id}/repair
Revive failed replicas without restarting healthy replicas.
GET /v1/pipeline_replica_sets/{id}/jobs
List active replica jobs.
Create request
Send the following fields to POST /v1/pipeline_replica_sets:
name
string
-
Must be non-empty.
query
string
-
Must be non-empty and compile to an HTTP-source pipeline.
replicas
u32
1
From 1 through 30.
parallelism
u64
1
At least 1 and no greater than the plan's maximum.
checkpointIntervalMicros
u64
10_000_000 (10 seconds)
At least 5_000_000 on create.
PATCH accepts any subset of replicas, parallelism, checkpointIntervalMicros, and stop. On patch, the checkpoint interval can range from 1 second through 1 day.
Phases
The PRS phase summarizes the state of its active replica jobs:
Created
The replica set exists but has no active jobs yet.
Scheduling
Jobs are starting and none is Running yet.
Running
Every active replica is Running.
Degraded
Some, but not all, active replicas are Running.
Failed
Every active replica is Failed.
Stopped
The set has had jobs, and all of them are stopped.
Degraded means the endpoint can continue serving through the remaining ready replicas, but it has less capacity and redundancy than requested. Operators should investigate the failed jobs and repair the set.
Scaling
Update replicas to change the number of independent jobs:
Scaling up schedules additional replicas. Scaling down stops excess replicas and removes them from the load-balanced endpoint. Each stopped replica completes according to the requested stop behavior.
Update parallelism when the downstream plan needs more or fewer subtasks:
Replica count and parallelism solve different problems: replicas scale the HTTP entry point and its failure isolation, while parallelism scales supported work inside each replica job.
Repair and restart
Use repair when healthy replicas should remain available:
Repair revives failed replicas until the requested replica count is restored. Healthy replicas are not restarted.
Use restart for a full rotation of the set:
Restart replaces healthy active replicas and also revives failed replicas. Passing {"force": true} skips the final checkpoint for the restarted jobs, so they recover from their latest previously successful checkpoints.
e6 Ingestion Engine does not automatically invoke repair or restart. A set remains Degraded or Failed until an authorized user or automation performs the appropriate action.
Endpoint lifecycle
Each PRS owns a dedicated load-balanced endpoint. It is created with the set, updated as replicas become ready or stop, and removed when the set is deleted.
All replica jobs must be terminal before deletion; otherwise the API returns 400. This ensures that jobs stop accepting traffic before their endpoint is removed.
Last updated
Was this helpful?