> For the complete documentation index, see [llms.txt](https://docs.e6data.com/ingestion-engine/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.e6data.com/ingestion-engine/architecture/pipeline-replica-sets.md).

# Pipeline replica sets

A **Pipeline Replica Set (PRS)** runs multiple independent jobs for one HTTP-source pipeline behind a single load-balanced endpoint. It provides one logical pipeline name, a declared replica count, and a stable address for HTTP producers.

Use a PRS when an HTTP ingestion endpoint needs more availability or request capacity than one pipeline job can provide.

## Why HTTP sources use replicas

Partitioned sources such as Kafka, Kinesis, and object storage divide their input among parallel subtasks. An HTTP-push source is different: data arrives at a network endpoint, so increasing operator parallelism does not create additional independently reachable endpoints.

For an HTTP-source pipeline:

* use **replicas** to add independently running HTTP endpoints behind one load balancer;
* use **parallelism** for downstream processing where the compiled plan supports it; and
* use a normal pipeline, rather than a PRS, for sources that already partition their input.

A PRS addresses three operational requirements:

1. **Ingestion capacity** - requests can be distributed across multiple healthy replicas.
2. **Endpoint availability** - one replica can restart or recover while other replicas continue accepting requests.
3. **Stable addressing** - producers send requests to one endpoint without tracking individual jobs.

{% hint style="info" %}
A PRS is supported only for pipelines with an HTTP source. Creating one for another source returns `400 - PipelineReplicaSet is only supported for HTTP source pipelines`.
{% endhint %}

## Architecture and failure model

Each replica is a complete pipeline job with its own runtime state, checkpoints, and recovery lifecycle. Replicas share the compiled pipeline definition and the load-balanced HTTP endpoint; they do not share in-memory operator state.

|                       | Single pipeline                                | Pipeline Replica Set                                 |
| --------------------- | ---------------------------------------------- | ---------------------------------------------------- |
| Unit of execution     | One job                                        | Multiple independent jobs                            |
| Scale control         | Operator parallelism                           | Replica count, plus supported downstream parallelism |
| HTTP endpoint         | One job endpoint                               | One endpoint load-balanced across healthy replicas   |
| Failure impact        | Endpoint is unavailable while the job recovers | Other healthy replicas continue serving requests     |
| Checkpoints and state | One recovery history                           | Independent recovery history per replica             |

In Kubernetes, one dedicated `Service` routes requests to the ready replicas in the set. Readiness controls prevent traffic from being sent to a replica that is starting, stopping, or recovering.

![Pipeline Replica Set architecture](https://1725648790-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuztQ31u5OyLCkPgUTosp%2Fuploads%2Fgit-blob-1ecf2e30802bf2c1f21c7f7e3a48615bb92b2739%2Farchitecture-pipeline-replica-sets-1.svg?alt=media)

Because replicas checkpoint independently, a failed replica recovers from its own latest successful checkpoint without rolling back healthy replicas. Producers should still use normal retry and idempotency practices because an individual request can fail while a replica is being removed from service.

## REST API

Authorized API clients can create and manage replica sets directly.

| Method and path                               | Purpose                                                               |
| --------------------------------------------- | --------------------------------------------------------------------- |
| `POST /v1/pipeline_replica_sets`              | Create a PRS, compile its query, and schedule the requested replicas. |
| `GET /v1/pipeline_replica_sets`               | List replica sets.                                                    |
| `GET /v1/pipeline_replica_sets/{id}`          | Get one PRS, its phase, and its active replica jobs.                  |
| `PATCH /v1/pipeline_replica_sets/{id}`        | Change replicas, parallelism, checkpoint interval, or stop state.     |
| `DELETE /v1/pipeline_replica_sets/{id}`       | Delete a PRS after all of its jobs are terminal.                      |
| `POST /v1/pipeline_replica_sets/{id}/restart` | Rotate healthy replicas and revive failed replicas.                   |
| `POST /v1/pipeline_replica_sets/{id}/repair`  | Revive failed replicas without restarting healthy replicas.           |
| `GET /v1/pipeline_replica_sets/{id}/jobs`     | List active replica jobs.                                             |

### Create request

Send the following fields to `POST /v1/pipeline_replica_sets`:

| Field                      | Type   | Default                   | Validation                                                |
| -------------------------- | ------ | ------------------------- | --------------------------------------------------------- |
| `name`                     | string | -                         | Must be non-empty.                                        |
| `query`                    | string | -                         | Must be non-empty and compile to an HTTP-source pipeline. |
| `replicas`                 | u32    | `1`                       | From `1` through `30`.                                    |
| `parallelism`              | u64    | `1`                       | At least `1` and no greater than the plan's maximum.      |
| `checkpointIntervalMicros` | u64    | `10_000_000` (10 seconds) | At least `5_000_000` on create.                           |

`PATCH` accepts any subset of `replicas`, `parallelism`, `checkpointIntervalMicros`, and `stop`. On patch, the checkpoint interval can range from 1 second through 1 day.

## Phases

The PRS phase summarizes the state of its active replica jobs:

| Phase        | Meaning                                            |
| ------------ | -------------------------------------------------- |
| `Created`    | The replica set exists but has no active jobs yet. |
| `Scheduling` | Jobs are starting and none is `Running` yet.       |
| `Running`    | Every active replica is `Running`.                 |
| `Degraded`   | Some, but not all, active replicas are `Running`.  |
| `Failed`     | Every active replica is `Failed`.                  |
| `Stopped`    | The set has had jobs, and all of them are stopped. |

`Degraded` means the endpoint can continue serving through the remaining ready replicas, but it has less capacity and redundancy than requested. Operators should investigate the failed jobs and repair the set.

## Scaling

Update `replicas` to change the number of independent jobs:

```json
{
  "replicas": 4
}
```

Scaling up schedules additional replicas. Scaling down stops excess replicas and removes them from the load-balanced endpoint. Each stopped replica completes according to the requested stop behavior.

Update `parallelism` when the downstream plan needs more or fewer subtasks:

```json
{
  "parallelism": 8
}
```

Replica count and parallelism solve different problems: replicas scale the HTTP entry point and its failure isolation, while parallelism scales supported work inside each replica job.

## Repair and restart

Use repair when healthy replicas should remain available:

```
POST /v1/pipeline_replica_sets/{id}/repair
```

Repair revives failed replicas until the requested replica count is restored. Healthy replicas are not restarted.

Use restart for a full rotation of the set:

```
POST /v1/pipeline_replica_sets/{id}/restart
```

Restart replaces healthy active replicas and also revives failed replicas. Passing `{"force": true}` skips the final checkpoint for the restarted jobs, so they recover from their latest previously successful checkpoints.

e6 Ingestion Engine does not automatically invoke repair or restart. A set remains `Degraded` or `Failed` until an authorized user or automation performs the appropriate action.

## Endpoint lifecycle

Each PRS owns a dedicated load-balanced endpoint. It is created with the set, updated as replicas become ready or stop, and removed when the set is deleted.

All replica jobs must be terminal before deletion; otherwise the API returns `400`. This ensures that jobs stop accepting traffic before their endpoint is removed.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.e6data.com/ingestion-engine/architecture/pipeline-replica-sets.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.