Limitations
What the e6data PySpark compatibility layer does not support, and how unsupported operations behave.
The e6-spark-compat library covers the PySpark DataFrame API and a large set of SQL functions, but it isn't a complete drop-in for every Spark feature. Know these boundaries before migrating a workload. For what is supported, see Supported APIs and compatibility notes.
Read-oriented engine
e6data is a query engine for reading data. The compatibility layer translates DataFrame operations into SQL that runs on e6data - it is not a general-purpose Spark runtime.
No RDD API. Only the DataFrame and SQL APIs are supported, not low-level RDD operations.
No UDFs / arbitrary Python execution. Operations must translate to SQL; arbitrary Python or JVM UDFs that run inside Spark executors aren't supported.
No Spark-managed cluster or streaming. There's no
SparkContext-style cluster management or Structured Streaming; the session connects to an e6data cluster.
Partial API coverage
Some APIs are supported only in part:
Window functions and complex types are fully available in standard execution but only a subset runs on the native execution path.
Geospatial functions are available via the Sedona-compatible layer in standard execution but not on the native path.
Cache hints (
cache(),persist(),coalesce()) are accepted as pass-through hints rather than Spark-style materialization.
When a query uses a feature the native path doesn't support, e6data falls back to the standard path automatically for that query - no error, no missed result.
Write operations
Write support depends on the target catalog and table format; e6data is read-only for some catalog types. Confirm write support for your catalog before relying on it.
See also
Last updated
Was this helpful?