Query e6data from Python

Before you start

You need:

A running Query Engine cluster. On CloudPrem, the cluster also needs ingress. See Configure cluster ingress.
A personal access token. See Create, rotate and revoke tokens.
Python 3, with a C++ compiler. On Windows, install Visual C++ Build Tools first (see Windows setup below).

Find the host, port and cluster name in the console under Cluster → Connectivity.

Install the connector

Install the package from PyPI.

terminal

pip install --no-cache-dir e6data-python-connector

On Amazon Linux, CentOS, Ubuntu or Debian, install the build dependencies first if the install fails:

yum install python3-devel gcc-c++ -y
pip install wheel

apt install python3-dev g++ -y
pip install wheel

Set your credentials

Set the cluster host, your e6data email and your access token as environment variables. Do not hardcode them in your script.

terminal

export E6_HOST="<cluster host>"
export E6_USER="[email protected]"
export E6_TOKEN="<access token>"

Connect and run a query

Open a connection to the cluster, create a cursor on your catalog, and run SQL.

query.py

import os
from e6data_python_connector import Connection

conn = Connection(
    host=os.getenv("E6_HOST"),
    port=443,                         # use the port shown under Cluster → Connectivity
    username=os.getenv("E6_USER"),    # your e6data email
    password=os.getenv("E6_TOKEN"),   # a personal access token
    database="<schema>",
    cluster_name="<cluster_name>",
    secure=True,                      # use TLS
)

cursor = conn.cursor(catalog_name="<catalog>")
query_id = cursor.execute("SELECT * FROM <table> LIMIT 10")
for row in cursor.fetchall():
    print(row)

Large results. Use cursor.fetchmany(500) to limit rows, or loop over cursor.fetchall_buffer() to stream rows without loading them all into memory.

Close the connection

Clear and close the cursor, then close the connection. This frees old results from memory.

query.py

cursor.clear()   # not needed if you aborted the query
cursor.close()
conn.close()

Windows setup

The connector uses a C-based deserialization library (fastbinary). On Windows it works only when Visual C++ Build Tools are installed. Without them you see this error:

ImportError: cannot import name 'fastbinary' from 'thrift.protocol'

To fix it:

Install Visual C++ Build Tools and select the Desktop Development with C++ workload. Launch the Build Tools once after installing.

Create and activate a virtual environment:

python -m venv <env_name>
<env_name>\Scripts\activate.bat

Install the connector in that environment:

pip install --no-cache-dir e6data-python-connector

If you can’t install the build tools, pass require_fastbinary=False to Connection. The connector then uses a pure-Python deserializer, which is about 2 to 3 times slower but otherwise works the same.

Python connector reference

Reference · Connection parameters and the cursor API.

REST API for SQL queries

How-to · A driverless alternative over HTTPS.

Personal access tokens

Platform · Tokens the connector signs in with.

Source code: e6data-python-connector on GitHub · Package: PyPI

Query Engine

Get started

Connect data

Run clusters

Query

Connect your tools

Secure and govern

Troubleshoot

Reference

Concepts

Query e6data from Python

Before you start

Windows setup

Python connector reference

REST API for SQL queries

Personal access tokens

​Before you start

​Windows setup

​Related

Python connector reference

REST API for SQL queries

Personal access tokens

Before you start

Windows setup

Related