> ## Documentation Index > Fetch the complete documentation index at: https://docs.e6data.com/llms.txt > Use this file to discover all available pages before exploring further. # Pipeline replica sets > A Pipeline Replica Set (PRS) runs multiple independent jobs for one HTTP-source pipeline behind a single load-balanced endpoint. It provides one logical pipeline name, a declared replica count, and a stable address for HTTP producers. Use a PRS when an HTTP ingestion endpoint needs more availability or request capacity than one pipeline job can provide. ## Why HTTP sources use replicas Partitioned sources such as Kafka, Kinesis, and object storage divide their input among parallel subtasks. An HTTP-push source is different: data arrives at a network endpoint, so increasing operator parallelism does not create additional independently reachable endpoints. For an HTTP-source pipeline: * use **replicas** to add independently running HTTP endpoints behind one load balancer; * use **parallelism** for downstream processing where the compiled plan supports it; and * use a normal pipeline, rather than a PRS, for sources that already partition their input. A PRS addresses three operational requirements: 1. **Ingestion capacity** - requests can be distributed across multiple healthy replicas. 2. **Endpoint availability** - one replica can restart or recover while other replicas continue accepting requests. 3. **Stable addressing** - producers send requests to one endpoint without tracking individual jobs. A PRS is supported only for pipelines with an HTTP source. Creating one for another source returns `400 - PipelineReplicaSet is only supported for HTTP source pipelines`. ## Architecture and failure model Each replica is a complete pipeline job with its own runtime state, checkpoints, and recovery lifecycle. Replicas share the compiled pipeline definition and the load-balanced HTTP endpoint; they do not share in-memory operator state. | | Single pipeline | Pipeline Replica Set | | - | - | - | | Unit of execution | One job | Multiple independent jobs | | Scale control | Operator parallelism | Replica count, plus supported downstream parallelism | | HTTP endpoint | One job endpoint | One endpoint load-balanced across healthy replicas | | Failure impact | Endpoint is unavailable while the job recovers | Other healthy replicas continue serving requests | | Checkpoints and state | One recovery history | Independent recovery history per replica | In Kubernetes, one dedicated `Service` routes requests to the ready replicas in the set. Readiness controls prevent traffic from being sent to a replica that is starting, stopping, or recovering. Pipeline Replica Set architecture Because replicas checkpoint independently, a failed replica recovers from its own latest successful checkpoint without rolling back healthy replicas. Producers should still use normal retry and idempotency practices because an individual request can fail while a replica is being removed from service. ## REST API Authorized API clients can create and manage replica sets directly. | Method and path | Purpose | | - | - | | `POST /api/v1/pipeline_replica_sets` | Create a PRS, compile its query, and schedule the requested replicas. | | `GET /api/v1/pipeline_replica_sets` | List replica sets. | | `GET /api/v1/pipeline_replica_sets/{id}` | Get one PRS, its phase, and its active replica jobs. | | `PATCH /api/v1/pipeline_replica_sets/{id}` | Change replicas, parallelism, checkpoint interval, or stop state. | | `DELETE /api/v1/pipeline_replica_sets/{id}` | Delete a PRS after all of its jobs are terminal. | | `POST /api/v1/pipeline_replica_sets/{id}/restart` | Rotate healthy replicas and revive failed replicas. | | `POST /api/v1/pipeline_replica_sets/{id}/repair` | Revive failed replicas without restarting healthy replicas. | | `GET /api/v1/pipeline_replica_sets/{id}/jobs` | List active replica jobs. | ### Create request Send the following fields to `POST /api/v1/pipeline_replica_sets`: | Field | Type | Default | Validation | | - | - | - | - | | `name` | string | - | Must be non-empty. | | `query` | string | - | Must be non-empty and compile to an HTTP-source pipeline. | | `replicas` | u32 | `1` | From `1` through `30`. | | `parallelism` | u64 | `1` | At least `1` and no greater than the plan's maximum. | | `checkpointIntervalMicros` | u64 | `10_000_000` (10 seconds) | At least `5_000_000` on create. | `PATCH` accepts any subset of `replicas`, `parallelism`, `checkpointIntervalMicros`, and `stop`. On patch, the checkpoint interval can range from 1 second through 1 day. ## Phases The PRS phase summarizes the state of its active replica jobs: | Phase | Meaning | | - | - | | `Created` | The replica set exists but has no active jobs yet. | | `Scheduling` | Jobs are starting and none is `Running` yet. | | `Running` | Every active replica is `Running`. | | `Degraded` | Some, but not all, active replicas are `Running`. | | `Failed` | Every active replica is `Failed`. | | `Stopped` | The set has had jobs, and all of them are stopped. | `Degraded` means the endpoint can continue serving through the remaining ready replicas, but it has less capacity and redundancy than requested. Operators should investigate the failed jobs and repair the set. ## Scaling Update `replicas` to change the number of independent jobs: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "replicas": 4 } ``` Scaling up schedules additional replicas. Scaling down stops excess replicas and removes them from the load-balanced endpoint. Each stopped replica completes according to the requested stop behavior. Update `parallelism` when the downstream plan needs more or fewer subtasks: ```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "parallelism": 8 } ``` Replica count and parallelism solve different problems: replicas scale the HTTP entry point and its failure isolation, while parallelism scales supported work inside each replica job. ## Repair and restart Use repair when healthy replicas should remain available: ``` POST /api/v1/pipeline_replica_sets/{id}/repair ``` Repair revives failed replicas until the requested replica count is restored. Healthy replicas are not restarted. Use restart for a full rotation of the set: ``` POST /api/v1/pipeline_replica_sets/{id}/restart ``` Restart replaces healthy active replicas and also revives failed replicas. Passing `{"force": true}` skips the final checkpoint for the restarted jobs, so they recover from their latest previously successful checkpoints. Ingest Engine does not automatically invoke repair or restart. A set remains `Degraded` or `Failed` until an authorized user or automation performs the appropriate action. ## Endpoint lifecycle Each PRS owns a dedicated load-balanced endpoint. It is created with the set, updated as replicas become ready or stop, and removed when the set is deleted. All replica jobs must be terminal before deletion; otherwise the API returns `400`. This ensures that jobs stop accepting traffic before their endpoint is removed. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.