vortex-data / vortex
An extensible, state of the art columnar file format. Formerly at @spiraldb, now an Incubation Stage project at LFAI&Data, part of the Linux Foundation.
View on GitHubAI Architecture Analysis
This repository is indexed by RepoMind. By analyzing vortex-data/vortex in our AI interface, you can instantly generate complete architecture diagrams, visualize control flows, and perform automated security audits across the entire codebase.
Our Agentic Context Augmented Generation (Agentic CAG) engine loads full source files into context on-demand, avoiding the fragmentation of traditional RAG systems. Ask questions about the architecture, dependencies, or specific features to see it in action.
Embed this Badge
Showcase RepoMind's analysis directly in your repository's README.
Repository Overview (README excerpt)
Crawler view🌪️ Vortex Join the community on Slack! | Documentation | Performance Benchmarks Overview Vortex is a next-generation columnar file format and toolkit designed for high-performance data processing. It is the fastest and most extensible format for building data systems backed by object storage. It provides: • **Blazing Fast Performance** • 100x faster random access reads (vs. modern Apache Parquet) • 10-20x faster scans • 5x faster writes • Similar compression ratios • Efficient support for wide tables with zero-copy/zero-parse metadata • **Extensible Architecture** • Modeled after Apache DataFusion's extensible approach • Pluggable encoding system, type system, compression strategy, & layout strategy • Zero-copy compatibility with Apache Arrow • **Open Source, Neutral Governance** • A Linux Foundation (LF AI & Data) Project • Apache-2.0 Licensed • **Integrations** • Arrow, DataFusion, DuckDB, Spark, Pandas, Polars, & more • Apache Iceberg (coming soon) > 🟢 **Development Status**: Library APIs may change from version to version, but we now consider > the file format _stable_ . From release 0.36.0, all future releases of Vortex should > maintain backwards compatibility of the file format (i.e., be able to read files written by > any earlier version >= 0.36.0). Key Features Core Capabilities • **Logical Types** - Clean separation between logical schema and physical layout • **Zero-Copy Arrow Integration** - Seamless conversion to/from Apache Arrow arrays • **Extensible Encodings** - Pluggable physical layouts with built-in optimizations • **Cascading Compression** - Support for nested encoding schemes • **High-Performance Computing** - Optimized compute kernels for encoded data • **Rich Statistics** - Lazy-loaded summary statistics for optimization Technical Architecture Logical vs Physical Design Vortex strictly separates logical and physical concerns: • **Logical Layer**: Defines data types and schema • **Physical Layer**: Handles encoding and storage implementation • **Built-in Encodings**: Compatible with Apache Arrow's memory format • **Extension Encodings**: Optimized compression schemes (RLE, dictionary, etc.) Quick Start Installation Rust Crate All features are exported through the main crate. Python Package Command Line UI (vx) For browsing the structure of Vortex files, you can use the command-line tool. Development Setup Prerequisites (macOS) Benchmarking Use to run benchmarks comparing engines (DataFusion, DuckDB) and formats (Parquet, Vortex): See bench-orchestrator/README.md for full documentation. Performance Optimization For optimal performance, we suggest using MiMalloc: Project Information License Licensed under the Apache License, Version 2.0. Governance Vortex is an independent open-source project and not controlled by any single company. The Vortex Project is a sub-project of the Linux Foundation Projects. The governance model is documented in CONTRIBUTING.md and is subject to the terms of the Technical Charter. Contributing Please **do** read CONTRIBUTING.md before you contribute. Reporting Vulnerabilities If you discover a security vulnerability, please email . Trademarks Copyright © Vortex a Series of LF Projects, LLC. For terms of use, trademark policy, and other project policies please see Acknowledgments The Vortex project benefits enormously from groundbreaking work from the academic & open-source communities. Research in Vortex • BtrBlocks - Efficient columnar compression • FastLanes & FastLanes on GPU - High-performance integer compression • FSST - Fast random access string compression • ALP & G-ALP - Adaptive lossless floating-point compression • Procella - YouTube's unified data system • Anyblob - High-performance access to object storage • ClickHouse - Fast analytics for everyone • MonetDB/X100 - Hyper-Pipelining Query Execution • Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Format for the Many-Core Age • The FastLanes File Format - Expression Operators Vortex in Research • Anyblox - A Framework for Self-Decoding Datasets • F3 - Open-Source Data File Format for the Future Open Source Inspiration • Apache Arrow • Apache DataFusion • parquet2 by Jorge Leitao • DuckDB • Velox & Nimble Thanks to all contributors who have shared their knowledge and code with the community! 🚀