Skip to main content
Use a PRS when an HTTP ingestion endpoint needs more availability or request capacity than one pipeline job can provide.

​
Why HTTP sources use replicas

Partitioned sources such as Kafka, Kinesis, and object storage divide their input among parallel subtasks. An HTTP-push source is different: data arrives at a network endpoint, so increasing operator parallelism does not create additional independently reachable endpoints. For an HTTP-source pipeline:
  • use replicas to add independently running HTTP endpoints behind one load balancer;
  • use parallelism for downstream processing where the compiled plan supports it; and
  • use a normal pipeline, rather than a PRS, for sources that already partition their input.
A PRS addresses three operational requirements:
  1. Ingestion capacity - requests can be distributed across multiple healthy replicas.
  2. Endpoint availability - one replica can restart or recover while other replicas continue accepting requests.
  3. Stable addressing - producers send requests to one endpoint without tracking individual jobs.
A PRS is supported only for pipelines with an HTTP source. Creating one for another source returns 400 - PipelineReplicaSet is only supported for HTTP source pipelines.

​
Architecture and failure model

Each replica is a complete pipeline job with its own runtime state, checkpoints, and recovery lifecycle. Replicas share the compiled pipeline definition and the load-balanced HTTP endpoint; they do not share in-memory operator state. In Kubernetes, one dedicated Service routes requests to the ready replicas in the set. Readiness controls prevent traffic from being sent to a replica that is starting, stopping, or recovering. Pipeline Replica Set architecture Because replicas checkpoint independently, a failed replica recovers from its own latest successful checkpoint without rolling back healthy replicas. Producers should still use normal retry and idempotency practices because an individual request can fail while a replica is being removed from service.

​
REST API

Authorized API clients can create and manage replica sets directly.

​
Create request

Send the following fields to POST /api/v1/pipeline_replica_sets: PATCH accepts any subset of replicas, parallelism, checkpointIntervalMicros, and stop. On patch, the checkpoint interval can range from 1 second through 1 day.

​
Phases

The PRS phase summarizes the state of its active replica jobs: Degraded means the endpoint can continue serving through the remaining ready replicas, but it has less capacity and redundancy than requested. Operators should investigate the failed jobs and repair the set.

​
Scaling

Update replicas to change the number of independent jobs:
Scaling up schedules additional replicas. Scaling down stops excess replicas and removes them from the load-balanced endpoint. Each stopped replica completes according to the requested stop behavior. Update parallelism when the downstream plan needs more or fewer subtasks:
Replica count and parallelism solve different problems: replicas scale the HTTP entry point and its failure isolation, while parallelism scales supported work inside each replica job.

​
Repair and restart

Use repair when healthy replicas should remain available:
Repair revives failed replicas until the requested replica count is restored. Healthy replicas are not restarted. Use restart for a full rotation of the set:
Restart replaces healthy active replicas and also revives failed replicas. Passing {"force": true} skips the final checkpoint for the restarted jobs, so they recover from their latest previously successful checkpoints. Ingest Engine does not automatically invoke repair or restart. A set remains Degraded or Failed until an authorized user or automation performs the appropriate action.

​
Endpoint lifecycle

Each PRS owns a dedicated load-balanced endpoint. It is created with the set, updated as replicas become ready or stop, and removed when the set is deleted. All replica jobs must be terminal before deletion; otherwise the API returns 400. This ensures that jobs stop accepting traffic before their endpoint is removed.
Last modified on October 1, 2026