TDengine into Apache Arrow with ADBC
TDengine 3.3.6, read and written through adbcBridge — the ADBC driver that loads the database's ODBC driver — read-only on Linux and macOS arm64. Everything below is the compatibility matrix's record for this row.
Status
PASS
PASS (3.3.6.13, vendor arm64 client)
FAIL, driver-side, the NCHAR column only (TDengine 3.3.6.13 server; taos_odbc from the Windows client package 3.4.2.5, TDengine driver). Two things had to change to connect at all: the 3.4.2.5 client cannot speak the native protocol to a 3.3.6 server (TCP connects, the server drops it with read invalid packet; not IPv6), so the entry runs over websocket through taosadapter — URL={ws://root:[email protected]:16041} with port 6041 published by a compose override — which reaches the server and passes every column but s. That column: taos_odbc takes the client character set from GetACP() (1252) and ignores CHARSET_FOR_COL_BIND/CHARSET_ENCODER_FOR_COL_BIND and a UCRT setlocale(".UTF-8") alike, so reading héllo 🚀 fails [iconv] Character set conversion for UTF-32LE to CP1252 failed for SQL_C_CHAR and SQL_C_WCHAR (pyodbc identical), and a SQL_C_BINARY read shows the value was already stored double-encoded on the way in. Only a UTF-8 system code page changes that. Fetch of the ASCII fixture works: 298,618 rows/s
Driver and connection string
- ODBC driver
- taos-odbc (TDengine's own connector, built from source)
- Wire
- native ODBC
- adbcBridge
- 0.1.1 (release)
The connection string the matrix used, as the connection strings reference records it (Databases with their own native protocol and ODBC driver); {drv} is the driver library or its name from odbcinst.ini, and the host, port and credentials are the test server's.
Driver={drv};SERVER=127.0.0.1:16030;UID=root;PWD=taosdata;TIMESTAMP_AS_IS=1;Python
import adbcbridge
# {drv}: the ODBC driver library (path), or its name from odbcinst.ini
conn = adbcbridge.connect(uri="Driver={drv};SERVER=127.0.0.1:16030;UID=root;PWD=taosdata;TIMESTAMP_AS_IS=1;")
with conn.cursor() as cur:
cur.execute("SELECT * FROM my_table")
table = cur.fetch_arrow_table() # pyarrow.Table
Polars and pandas
import polars as pl, pandas as pd
df = pl.read_database("SELECT * FROM my_table", connection=conn) # Polars, Arrow-native
pdf = pd.read_sql("SELECT * FROM my_table", conn) # pandas 2.2+, ADBC connection
pip install adbcbridge brings the driver library; the TDengine ODBC driver is installed the way its vendor documents, then named in Driver=. Rust, Go, C# and Java use the same connection string through their ADBC driver managers — see the docs.
What the matrix recorded
time-series server whose every table must start with a TIMESTAMP primary key that is non-NULL, distinct and inside the retention window, so no generated ingest DDL and no positional workload INSERT fits one -- the entry reads tables its setup builds and bulk-ingests through extra into a timestamp-first table; driver quirks handled: no SQL_C_TYPE_TIMESTAMP conversion for bound columns or parameters (SQLGetData does convert; the bridge reads through bound block cursors, so timestamp columns are read as text and timestamp params sent as text), a boolean param is taken only as SQL_C_SBIGINT described as SQL_TINYINT, the driver implements no DECIMAL (a DECIMAL column fails the whole SELECT with not supported yet) so that column is exact text; backtick identifiers; fetch 403k rows/s