> ## Documentation Index
> Fetch the complete documentation index at: https://docs.e6data.com/llms.txt
> Use this file to discover all available pages before exploring further.

# PySpark compatibility

> Run existing PySpark and Apache Sedona code on e6data with a drop-in compatibility library - install, configure the connection, and run.

`e6-spark-compat` is a drop-in compatibility library that lets you run existing PySpark and Apache Sedona code on e6data. Update your import statements, configure the e6data connection, and your Spark code works as-is - no rewrites needed.

DataFrame operations are lazily evaluated: transformations build a query-plan tree, and when an action (`collect`, `show`, `count`) is called, the plan is translated into optimized SQL and executed on e6data.

## Key capabilities

* Full PySpark DataFrame API - `select`, `filter`, `join`, `groupBy`, `orderBy`, `union`, `pivot`, and more.
* 130+ SQL functions - string, math, aggregate, date/time, window, conditional.
* Window functions with the complete `Window` specification API.
* 70+ Apache Sedona-compatible spatial functions (`ST_*`).
* File format support - Parquet, ORC, CSV, JSON, GeoParquet, Delta.
* Read and write operations.

See [Supported APIs and compatibility notes](/query/pyspark/supported-apis) for the full list, and [Limitations](/query/pyspark/limitations) for what isn't supported.

## Prerequisites

* An active e6data workspace and cluster.
* A [personal access token](/platform/tokens/pat-and-service-account-keys) from the e6data Console (**User settings → Access tokens**).
* Python 3.8+.

## Installation

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
# Install from PyPI
pip install e6data-spark-compatibility

# With spatial support
pip install e6data-spark-compatibility[spatial]
```

## Getting started

Migrate a PySpark application to e6data in three steps: update imports, configure the connection, and run.

### Step 1: Update imports

Replace your PySpark imports with `e6-spark-compat` equivalents. The API is identical.

<Tabs>
  <Tab title="Before (PySpark)">
    ```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    from pyspark.sql import SparkSession
    from pyspark.sql.functions import col, upper, count, sum, row_number
    from pyspark.sql.window import Window
    ```
  </Tab>

  <Tab title="After (e6data)">
    ```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
    from e6_spark_compat import SparkSession
    from e6_spark_compat.sql.functions import col, upper, count, sum, row_number
    from e6_spark_compat.sql.window import Window
    ```
  </Tab>
</Tabs>

For spatial (Sedona) operations, replace `from sedona.register import SedonaRegistrator` with `from e6_spark_compat.sedona import SedonaRegistrator`.

### Step 2: Configure the connection

Create a `SparkSession` pointing to your e6data cluster.

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
spark = (SparkSession.builder
    .appName("MyApp")
    .config("spark.e6data.host", "<cluster-host>")
    .config("spark.e6data.username", "<username>")
    .config("spark.e6data.password", "<access-token>")
    .config("spark.e6data.database", "<database>")
    .config("spark.e6data.catalog", "<catalog>")
    .config("spark.e6data.cluster", "<cluster-name>")
    .config("spark.e6data.secure", True)
    .getOrCreate())
```

| Parameter | Description | Required |
| - | - | - |
| `spark.e6data.host` | Cluster hostname or IP address | Yes |
| `spark.e6data.username` | e6data account email | Yes |
| `spark.e6data.password` | Personal access token from the e6data Console | Yes |
| `spark.e6data.database` | Target database name | Yes |
| `spark.e6data.catalog` | Catalog name | Yes |
| `spark.e6data.cluster` | Cluster name | Yes |
| `spark.e6data.secure` | Use TLS (`True` or `False`). Default `True` | No |

<Info>
  Find your cluster hostname and connection details in the e6data Console under **Clusters → Connection Info**.
</Info>

<Warning>
  Don't hardcode your access token in source code. Use environment variables or a secrets manager - for example, `os.getenv("E6DATA_TOKEN")`.
</Warning>

### Step 3: Run your code

Your existing PySpark logic works without modification:

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
df = spark.read.parquet("s3://bucket/path/to/data.parquet")

result = (df.filter(col("age") > 21)
    .select("name", "city", "salary")
    .groupBy("city")
    .agg(count("*").alias("total"), sum("salary").alias("total_salary"))
    .orderBy(col("total").desc()))

result.show()   # action triggers execution on e6data
```

## Catalog operations

Discover databases, tables, and columns programmatically:

```python theme={"theme":{"light":"github-light","dark":"github-dark"}}
spark.catalog.listDatabases()
spark.catalog.listTables()
spark.catalog.listColumns("my_table")
spark.catalog.tableExists("my_table")
```

Close the session with `spark.stop()`.

## See also

* [Supported APIs and compatibility notes](/query/pyspark/supported-apis)
* [Limitations](/query/pyspark/limitations)
* [Access tokens](/platform/tokens/pat-and-service-account-keys)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.