PySpark compatibility

e6-spark-compat is a drop-in compatibility library that lets you run existing PySpark and Apache Sedona code on e6data. Update your import statements, configure the e6data connection, and your Spark code works as-is - no rewrites needed. DataFrame operations are lazily evaluated: transformations build a query-plan tree, and when an action (collect, show, count) is called, the plan is translated into optimized SQL and executed on e6data.

Key capabilities

Full PySpark DataFrame API - select, filter, join, groupBy, orderBy, union, pivot, and more.
130+ SQL functions - string, math, aggregate, date/time, window, conditional.
Window functions with the complete Window specification API.
70+ Apache Sedona-compatible spatial functions (ST_*).
File format support - Parquet, ORC, CSV, JSON, GeoParquet, Delta.
Read and write operations.

See Supported APIs and compatibility notes for the full list, and Limitations for what isn’t supported.

Prerequisites

An active e6data workspace and cluster.
A personal access token from the e6data Console (User settings → Access tokens).
Python 3.8+.

Installation

# Install from PyPI
pip install e6data-spark-compatibility

# With spatial support
pip install e6data-spark-compatibility[spatial]

Getting started

Migrate a PySpark application to e6data in three steps: update imports, configure the connection, and run.

Step 1: Update imports

Replace your PySpark imports with e6-spark-compat equivalents. The API is identical.

Before (PySpark)
After (e6data)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, upper, count, sum, row_number
from pyspark.sql.window import Window

from e6_spark_compat import SparkSession
from e6_spark_compat.sql.functions import col, upper, count, sum, row_number
from e6_spark_compat.sql.window import Window

For spatial (Sedona) operations, replace from sedona.register import SedonaRegistrator with from e6_spark_compat.sedona import SedonaRegistrator.

Step 2: Configure the connection

Create a SparkSession pointing to your e6data cluster.

spark = (SparkSession.builder
    .appName("MyApp")
    .config("spark.e6data.host", "<cluster-host>")
    .config("spark.e6data.username", "<username>")
    .config("spark.e6data.password", "<access-token>")
    .config("spark.e6data.database", "<database>")
    .config("spark.e6data.catalog", "<catalog>")
    .config("spark.e6data.cluster", "<cluster-name>")
    .config("spark.e6data.secure", True)
    .getOrCreate())

Parameter	Description	Required
`spark.e6data.host`	Cluster hostname or IP address	Yes
`spark.e6data.username`	e6data account email	Yes
`spark.e6data.password`	Personal access token from the e6data Console	Yes
`spark.e6data.database`	Target database name	Yes
`spark.e6data.catalog`	Catalog name	Yes
`spark.e6data.cluster`	Cluster name	Yes
`spark.e6data.secure`	Use TLS (`True` or `False`). Default `True`	No

Find your cluster hostname and connection details in the e6data Console under Clusters → Connection Info.

Don’t hardcode your access token in source code. Use environment variables or a secrets manager - for example, os.getenv("E6DATA_TOKEN").

Step 3: Run your code

Your existing PySpark logic works without modification:

df = spark.read.parquet("s3://bucket/path/to/data.parquet")

result = (df.filter(col("age") > 21)
    .select("name", "city", "salary")
    .groupBy("city")
    .agg(count("*").alias("total"), sum("salary").alias("total_salary"))
    .orderBy(col("total").desc()))

result.show()   # action triggers execution on e6data

Catalog operations

Discover databases, tables, and columns programmatically:

spark.catalog.listDatabases()
spark.catalog.listTables()
spark.catalog.listColumns("my_table")
spark.catalog.tableExists("my_table")

Close the session with spark.stop().

Query Engine

Get started

Connect data

Run clusters

Query

Connect your tools

Secure and govern

Troubleshoot

Reference

Concepts

PySpark compatibility

Key capabilities

Prerequisites

Installation

Getting started

Step 1: Update imports

Step 2: Configure the connection

Step 3: Run your code

Catalog operations

See also

​Key capabilities

​Prerequisites

​Installation

​Getting started

​Step 1: Update imports

​Step 2: Configure the connection

​Step 3: Run your code

​Catalog operations

​See also

Key capabilities

Prerequisites

Installation

Getting started

Step 1: Update imports

Step 2: Configure the connection

Step 3: Run your code

Catalog operations

See also