Database · Flight SQL

Dremio into Apache Arrow with ADBC

Dremio 26.0.5 (OSS, Arrow Flight SQL), read and written through adbcBridge — the ADBC driver that loads the database's ODBC driver — read-only on Linux, macOS arm64 and Windows x64. Everything below is the compatibility matrix's record for this row.

Status

Linux✓ read

PASS

macOS arm64✓ read

PASS (Dremio Server (via ODBC) 26.00.0005; read-only) through a bridge built against iODBC 3.52.16 — Python fetch 1,341,476 rows/s. Through unixODBC: the same driver-manager abort as Flight SQL (the macOS arm64 build is the iODBC-width one, dremio/warpdrive#16; the Linux x86_64 build is 2-byte SQLWCHAR and returns its diagnostics normally under unixODBC 2.3.12)

Windows x64✓ read

PASS (Dremio 26.0.0005 through the same Arrow Flight SQL ODBC Windows build) — with the same text_as_binary fix as flightsql (it failed the astral check the same way, U+1F680 → U+F680, before it); 1,002,050 rows/s. Admin user created over PUT /apiv2/bootstrap/firstuser; the first compat run right after boot hit a $scratch metadata race (Object 'adbc_t' not found immediately after CTAS) and passed on rerun

Driver and connection string

ODBC driver: Flight SQL ODBC 0.9.7 (Dremio, open source)
Wire: Flight SQL
adbcBridge: 0.1.1 (release)

The connection string the matrix used, as the connection strings reference records it (Arrow Flight SQL); {drv} is the driver library or its name from odbcinst.ini, and the host, port and credentials are the test server's.

Driver={drv};Host=127.0.0.1;Port=32010;UID=adbc;PWD=Adbc2026pass;useEncryption=false;schema=$scratch;

Python

import adbcbridge

# {drv}: the ODBC driver library (path), or its name from odbcinst.ini
conn = adbcbridge.connect(uri="Driver={drv};Host=127.0.0.1;Port=32010;UID=adbc;PWD=Adbc2026pass;useEncryption=false;schema=$scratch;")
with conn.cursor() as cur:
    cur.execute("SELECT * FROM my_table")
    table = cur.fetch_arrow_table()      # pyarrow.Table

Polars and pandas

import polars as pl, pandas as pd

df = pl.read_database("SELECT * FROM my_table", connection=conn)   # Polars, Arrow-native
pdf = pd.read_sql("SELECT * FROM my_table", conn)                   # pandas 2.2+, ADBC connection

pip install adbcbridge brings the driver library; the Dremio ODBC driver is installed the way its vendor documents, then named in Driver=. Rust, Go, C# and Java use the same connection string through their ADBC driver managers — see the docs.

What the matrix recorded

the engine that publishes this driver, and it needed no new driver quirk — no_sql_columns is keyed on the driver name, so Dremio takes the path shared with sqlflite and InfluxDB, though against Dremio 26 SQLColumns does not in fact crash (it describes its 18 result columns and fetches every row), so here the quirk costs a describe rather than avoiding a segfault; read-only because of the driver alone -- Dremio writes fine ($scratch, the writable source a stock dremio-oss ships, takes CTAS, and a table created there with a column list is Iceberg and takes INSERT), but SQLBindParameter is HYC00 Unsupported function even after a SQLPrepare that succeeds, and SQLNumParams reports 0, so no parameter can reach the server and setup builds both tables with literal CREATE TABLE IF NOT EXISTS ... AS SELECT; unlike sqlflite the driver reports a DECIMAL's declared scale — described as (19, *s*), precision still always 19 — so decimals arrive exact in a wider decimal128 rather than as text; a string literal holding any character outside ISO-8859-1 (a BMP 漢 as much as an emoji) needs the _UTF8'…' prefix or Dremio's parser encodes it as ISO-8859-1 and fails planning; server side, the first admin user has to be created over the REST API before any login works, and the query context comes from a schema connection property the driver forwards as a gRPC header; fetch 1.1M rows/s