apache / auron

Apache Auron (Incubating) The Auron accelerator for big data engines (e.g., Spark, Flink) leverages native vectorized execution to accelerate query processing. It combines the power of the Apache DataFusion library and the scale of the distributed computing framework. Auron takes a fully optimized physical plan from a distributed computing framework, mapping it into DataFusion's execution plan, and performs native plan computation. The key capabilities of Auron include: • **Native execution**: Implemented in Rust, eliminating JVM overhead and enabling predictable performance. • **Vectorized computation**: Built on Apache Arrow's columnar format, fully leveraging SIMD instructions for batch processing. • **Pluggable architecture**: Seamlessly integrates with Apache Spark while designed for future extensibility to other engines. • **Production-hardened optimizations**: Multi-level memory management, compacted shuffle formats, and adaptive execution strategies developed through large-scale deployment. Based on the inherent well-defined extensibility of DataFusion, Auron can be easily extended to support: • Various object stores. • Operators. • Simple and Aggregate functions. • File formats. We encourage you to extend DataFusion capability directly and add support in Auron with simple modifications in plan-serde and extension translation. Build from source To build Auron from source, follow the steps below: • Install Rust Auron's native execution lib is written in Rust. You need to install Rust (nightly) before compiling. We recommend using rustup for installation. • Install JDK Auron is regularly tested on JDK 8, 11, and 17, and is also tested on JDK 21. Make sure is properly set and points to your desired version. • Check out the source code. • Build the project. You can build Auron either *locally* or *inside Docker* using one of the supported OS images via the unified script: . Run to see all available options. After the build completes, a fat JAR with all dependencies will be generated in either the directory (for local builds) or directory (for Docker builds), depending on the selected build mode. Run Spark Job with Auron Accelerator This section describes how to submit and configure a Spark Job with Auron support. • Move the Auron JAR to the Spark client classpath (normally spark-xx.xx.xx/jars/). • Add the following configs to spark configuration in : • submit a query with spark-sql, or other tools like spark-thriftserver: Performance TPC-DS 1TB Benchmark Results: For methodology and additional results, please refer to benchmark documentation. We also encourage you to benchmark Auron and share the results with us. 🤗 Community Subscribe Mailing Lists Mail List is the most recognized form of communication in the Apache community. Contact us through the following mailing list. | Name | Scope | | | |:-----------------------------------------------------------|:--------------------------------|:---------------------------------------------------------|:--------------------------------------------------------------| | dev@auron.apache.org | Development-related discussions | Subscribe | Unsubscribe | Contributing Interested in contributing to Auron? Please read our Contributing Guide for detailed information on how to get started. License Auron is licensed under the Apache 2.0 License. A copy of the license can be found here.