PySpark compatibility

Run existing PySpark and Apache Sedona code on e6data with a drop-in compatibility library - install, configure the connection, and run.

e6-spark-compat is a drop-in compatibility library that lets you run existing PySpark and Apache Sedona code on e6data. Update your import statements, configure the e6data connection, and your Spark code works as-is - no rewrites needed.

DataFrame operations are lazily evaluated: transformations build a query-plan tree, and when an action (collect, show, count) is called, the plan is translated into optimized SQL and executed on e6data.

Key capabilities

• Full PySpark DataFrame API - select, filter, join, groupBy, orderBy, union, pivot, and more.
• 130+ SQL functions - string, math, aggregate, date/time, window, conditional.
• Window functions with the complete Window specification API.
• 70+ Apache Sedona-compatible spatial functions (ST_*).
• File format support - Parquet, ORC, CSV, JSON, GeoParquet, Delta.
• Read and write operations.

See Supported APIs and compatibility notes for the full list, and Limitations for what isn't supported.

Prerequisites

• An active e6data workspace and cluster.
• A personal access token from the e6data Console (User settings → Access tokens).
• Python 3.8+.

Installation

# Install from PyPI
pip install e6data-spark-compatibility

# With spatial support
pip install e6data-spark-compatibility[spatial]

Getting started

Migrate a PySpark application to e6data in three steps: update imports, configure the connection, and run.

Step 1: Update imports

Replace your PySpark imports with e6-spark-compat equivalents. The API is identical.

Before (PySpark)

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, upper, count, sum, row_number
from pyspark.sql.window import Window

After (e6data)

from e6_spark_compat import SparkSession
from e6_spark_compat.sql.functions import col, upper, count, sum, row_number
from e6_spark_compat.sql.window import Window

For spatial (Sedona) operations, replace from sedona.register import SedonaRegistrator with from e6_spark_compat.sedona import SedonaRegistrator.

Step 2: Configure the connection

Create a SparkSession pointing to your e6data cluster.

spark = (SparkSession.builder
    .appName("MyApp")
    .config("spark.e6data.host", "<cluster-host>")
    .config("spark.e6data.username", "<username>")
    .config("spark.e6data.password", "<access-token>")
    .config("spark.e6data.database", "<database>")
    .config("spark.e6data.catalog", "<catalog>")
    .config("spark.e6data.cluster", "<cluster-name>")
    .config("spark.e6data.secure", True)
    .getOrCreate())

Parameter	Description	Required
spark.e6data.host	Cluster hostname or IP address	Yes
spark.e6data.username	e6data account email	Yes
spark.e6data.password	Personal access token from the e6data Console	Yes
spark.e6data.database	Target database name	Yes
spark.e6data.catalog	Catalog name	Yes
spark.e6data.cluster	Cluster name	Yes
spark.e6data.secure	Use TLS (True or False). Default True	No

Find your cluster hostname and connection details in the e6data Console under Clusters → Connection Info.

Don't hardcode your access token in source code. Use environment variables or a secrets manager - for example, os.getenv("E6DATA_TOKEN").

Step 3: Run your code

Your existing PySpark logic works without modification:

df = spark.read.parquet("s3://bucket/path/to/data.parquet")

result = (df.filter(col("age") > 21)
    .select("name", "city", "salary")
    .groupBy("city")
    .agg(count("*").alias("total"), sum("salary").alias("total_salary"))
    .orderBy(col("total").desc()))

result.show()   # action triggers execution on e6data

Catalog operations

Discover databases, tables, and columns programmatically:

spark.catalog.listDatabases()
spark.catalog.listTables()
spark.catalog.listColumns("my_table")
spark.catalog.tableExists("my_table")

Close the session with spark.stop().

See also

• Supported APIs and compatibility notes
• Limitations
• Access tokens