dgraph-io / badger

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast, distributed graph database. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB. Project Status Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style bank test runs nightly for 8h, with flag and ensures the maintenance of transactional guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure persistence and consistency. Badger is being used by a number of projects which includes Dgraph, Jaeger Tracing, UsenetExpress, and many more. The list of projects using Badger can be found here. Please consult the [Changelog] for more detailed information on releases. Note: Badger is built with go 1.23 and we refrain from bumping this version to minimize downstream effects of those using Badger in applications built with older versions of Go. [Changelog]: https://github.com/dgraph-io/badger/blob/main/CHANGELOG.md Table of Contents • BadgerDB • Project Status • Table of Contents • Getting Started • Installing • Installing Badger Command Line Tool • Choosing a version • Badger Documentation • Resources • Blog Posts • Design • Comparisons • Benchmarks • Projects Using Badger • Contributing • Contact Getting Started Installing To start using Badger, install Go 1.23 or above. Badger v3 and above needs go modules. From your project, run the following command This will retrieve the library. Installing Badger Command Line Tool Badger provides a CLI tool which can perform certain operations like offline backup/restore. To install the Badger CLI, retrieve the repository and checkout the desired version. Then run This will install the badger command line utility into your $GOBIN path. Badger Documentation Badger Documentation is available at https://badger.dgraph.io Resources Blog Posts • Introducing Badger: A fast key-value store written natively in Go • Make Badger crash resilient with ALICE • Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go • Concurrent ACID Transactions in Badger Design Badger was written with these design goals in mind: • Write a key-value database in pure Go. • Use latest research to build the fastest KV database for data sets spanning terabytes. • Optimize for SSDs. Badger’s design is based on a paper titled _[WiscKey: Separating Keys from Values in SSD-conscious Storage][wisckey]_. [wisckey]: https://www.usenix.org/system/files/conference/fast16/fast16-papers-lu.pdf Comparisons | Feature | Badger | RocksDB | BoltDB | | ----------------------------- | ------------------------------------------ | ----------------------------- | --------- | | Design | LSM tree with value log | LSM tree only | B+ tree | | High Read throughput | Yes | No | Yes | | High Write throughput | Yes | Yes | No | | Designed for SSDs | Yes (with latest research 1 ) | Not specifically 2 | No | | Embeddable | Yes | Yes | Yes | | Sorted KV access | Yes | Yes | Yes | | Pure Go (no Cgo) | Yes | No | Yes | | Transactions | Yes, ACID, concurrent with SSI 3 | Yes (but non-ACID) | Yes, ACID | | Snapshots | Yes | Yes | Yes | | TTL support | Yes | Yes | No | | 3D access (key-value-version) | Yes 4 | No | No | 1 The [WISCKEY paper][wisckey] (on which Badger is based) saw big wins with separating values from keys, significantly reducing the write amplification compared to a typical LSM tree. 2 RocksDB is an SSD optimized version of LevelDB, which was designed specifically for rotating disks. As such RocksDB's design isn't aimed at SSDs. 3 SSI: Serializable Snapshot Isolation. For more details, see the blog post Concurrent ACID Transactions in Badger 4 Badger provides direct access to value versions via its Iterator API. Users can also specify how many versions to keep per key via Options. Benchmarks We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the detailed logs for the benchmarks can be found in the [badger-bench] repo. More explanation, including graphs can be found the blog posts (linked above). [badger-bench]: https://github.com/dgraph-io/badger-bench Projects Using Badger Below is a list of known projects that use Badger: • Dgraph - Distributed graph database. • Jaeger - Distributed tracing platform. • go-ipfs - Go client for the InterPlanetary File System (IPFS), a new hypermedia distribution protocol. • Riot - An open-source, distributed search engine. • emitter - Scalable, low latency, distributed pub/sub broker with message storage, uses MQTT, gossip and badger. • OctoSQL - Query tool that allows you to join, analyse and transform data from multiple databases using SQL. • Dkron - Distributed, fault tolerant job scheduling system. • smallstep/certificates - Step-ca is an online certificate authority for secure, automated certificate management. • Sandglass - distributed, horizontally scalable, persistent, time sorted message queue. • TalariaDB - Grab's Distributed, low latency time-series database. • Sloop - Salesforce's Kubernetes History Visualization Project. • Usenet Express - Serving over 300TB of data with Badger. • gorush - A push notification server written in Go. • 0-stor - Single device object store. • Dispatch Protocol - Blockchain protocol for distributed application data analytics. • GarageMQ - AMQP server written in Go. • RedixDB - A real-time persistent key-value store with the same redis protocol. • BBVA - Raft backend implementation using BadgerDB for Hashicorp raft. • Fantom - aBFT Consensus platform for distributed applications. • decred - An open, progressive, and self-funding cryptocurrency with a system of community-based governance integrated into its blockchain. • OpenNetSys - Create useful dApps in any software language. • Hon…