Database · Flight SQL

Arrow Flight SQL into Apache Arrow with ADBC

Arrow Flight SQL (sqlflite 1.5.5 / DuckDB 1.1.1), read and written through adbcBridge — the ADBC driver that loads the database's ODBC driver — read-only on Linux, macOS arm64 and Windows x64. Everything below is the compatibility matrix's record for this row.

Status

Linux✓ read

PASS

macOS arm64✓ read

PASS (sqlflite (via ODBC) 00.00.0000, DuckDB 1.1.1; read-only) through a bridge built against iODBC 3.52.16 — Python fetch 8,260,509 rows/s. Through unixODBC it aborts at the first failing statement: the driver is built to iODBC's 4-byte SQLWCHAR, and unixODBC 2.3.12's driver manager overflows its own stack buffer (SQLWCHAR sqlstate[6] in extract_diag_error_w) reading the wide diagnostic on the first SQL_ERROR → __stack_chk_fail → SIGABRT with no message. Not the driver crashing, not the bridge: unixODBC's isql dies the same way, and through iODBC the same programs get the proper diagnostic

Windows x64✓ read

PASS (sqlflite 1.5.5 / DuckDB 1.1.1; arrow-flight-sql-odbc LATEST-win64, 00.09.0007) — with the text_as_binary fix in this tree: the driver's Windows build returns U+1F680 as U+F680 (the low 16 bits) through SQL_C_WCHAR and ? through SQL_C_CHAR (the ANSI code page), pyodbc identical, so the first run failed the astral check; its SQL_C_BINARY conversion of a text column hands the server's UTF-8 through byte-exact (probed with pyodbc), and the reader now takes that route for this driver on Windows. It is also the faster route — 5,211,319 rows/s against 2,956,638 wide, the fastest fetch of the Windows column

Driver and connection string

ODBC driver: Flight SQL ODBC 0.9.7 (Dremio, open source)
Wire: Flight SQL
adbcBridge: 0.1.1 (release)

The connection string the matrix used, as the connection strings reference records it (Arrow Flight SQL); {drv} is the driver library or its name from odbcinst.ini, and the host, port and credentials are the test server's.

Driver={drv};Host=127.0.0.1;Port=31337;UID=sqlflite_username;PWD=adbc;useEncryption=false;

Python

import adbcbridge

# {drv}: the ODBC driver library (path), or its name from odbcinst.ini
conn = adbcbridge.connect(uri="Driver={drv};Host=127.0.0.1;Port=31337;UID=sqlflite_username;PWD=adbc;useEncryption=false;")
with conn.cursor() as cur:
    cur.execute("SELECT * FROM my_table")
    table = cur.fetch_arrow_table()      # pyarrow.Table

Polars and pandas

import polars as pl, pandas as pd

df = pl.read_database("SELECT * FROM my_table", connection=conn)   # Polars, Arrow-native
pdf = pd.read_sql("SELECT * FROM my_table", conn)                   # pandas 2.2+, ADBC connection

pip install adbcbridge brings the driver library; the Arrow Flight SQL ODBC driver is installed the way its vendor documents, then named in Driver=. Rust, Go, C# and Java use the same connection string through their ADBC driver managers — see the docs.

What the matrix recorded

driver has no SQLBindParameter at all, so nothing can be written through it; driver quirks handled: SQLColumns segfaults on the first SQLFetch, so GetObjects describes a zero-row SELECT instead (no_sql_columns); every DECIMAL described as (19,0), so decimals are read as exact text; fetch 1.32M rows/s