datafusion-contrib / datafusion-table-providers

DataFusion Table Providers Note: This is not an official Apache Software Foundation project. The goal of this repo is to extend the capabilities of DataFusion to support additional data sources via implementations of the trait. Many of the table providers in this repo are for querying data from other database systems. Those providers also integrate with the crate to allow for more efficient query execution, such as pushing down joins between multiple tables from the same database system, or efficiently implementing TopK style queries ( ). To use these table providers with efficient federation push-down, add the crate and create a DataFusion using the Federation optimizer rule and query planner with: Table Providers • PostgreSQL • MySQL • SQLite • ClickHouse • DuckDB • Flight SQL • ODBC Development During development, and especially before opening a PR, it is recommended to run: This verifies that all features and all crates compile without building binaries. It’s much faster than cargo build and avoids issues with native/shared library dependencies and heavy compilation. Examples (in Rust) Run the included examples to see how to use the table providers: DuckDB SQLite Postgres In order to run the Postgres example, you need to have a Postgres server running. You can use the following command to start a Postgres server in a Docker container the example can use: ClickHouse In order to run the Clickhouse example, you need to have a Clickhouse server running. You can use the following command to start a Clickhouse server in a Docker container the example can use: MySQL In order to run the MySQL example, you need to have a MySQL server running. You can use the following command to start a MySQL server in a Docker container the example can use: Flight SQL ODBC ARM Mac Please see https://github.com/pacman82/odbc-api#os-x-arm--mac-m1 for reference. Steps: • Install unixodbc and sqliteodbc by . • Find local sqliteodbc driver path by running . The path might look like . • Set up odbc config file at with your local sqliteodbc path. Example config file: • Test configuration by running . If the path is printed out correctly, then you are all set. Examples (in Python) • Start a Python venv • Enter into venv • Inside python/ folder, run . • Inside python/examples/ folder, run the corresponding test using .