dtprj / dongting

中文 Table of Contents • Introduce • 10X Throughput • Zero Dependencies • Try it • Build • Run server • Run benchmark • Client usage • Client examples • Server configuration • Cluster management • Configure a multi-node cluster • Run admin tools • Cluster changes • Advanced • Import project to IDE • Build raft server through code • Trim JRE with JLink • About me Introduce The Dongting project is a high-performance engine that integrates RAFT, configuration server, messaging queues. Features are as follows: • **Multi RAFT group support**: Running multiple RAFT groups within a same process. Dynamic addition, removal, and updating of RAFT groups (sharding), allowing your cluster to scale dynamically. The state machine runs in the raft framework can be customized. • **Distributed configuration server DtKV**: Tree-based structure, supports linear consistency general K/V operations, watch, TTL expiration, and distributed lock, similar to etcd. • is an in-memory database, so the total data size cannot be too large, but it uses raft log as redo log, creates snapshots periodically, and will not lose a single record even in power failure. • Natively supports tree-like directories, the complexity of many operations is O(1), such as watching directories, very efficient. • Supports temporary directories, which will be automatically deleted as a whole after TTL expires. The deletion operation is atomic. • Does not support transactions, but provides CAS and very easy-to-use distributed locks. • **(Planned) MQ (message queues)**: Use RAFT log as message queue log. 10X Throughput Dongting is developed using performance-oriented programming. In simple tests where both server and client run on the same machine, modern high-performance PCs using server default settings can easily achieve over 1 million TPS with the benchmark program, and RT won't be too large either. Zero Dependencies Are you still troubled by dependency management issues that spread like wildfire? Have you ever been ridiculed for a 1GB image size? You have been saved now! Dongting has no dependencies. It does not rely on any third-party jar files (only SLF4J is optional, if it is not in the classpath, the project will use JDK logging). Dongting core has only two JAR packages, client and server together are less than 1MB, you can easily embed it into your program, whether it is client or server. Dongting does not place high demands on your JDK: client only requires Java 8, server only requires Java 11. Dongting does not require using high-performance hardware, such as RDMA or Optane, and it does not rely on any third-party services such as storage services provided by Amazon or other cloud service providers. It can even run well on HDD disks and Raspberry Pi. Dongting does not require you to adjust Linux kernel parameters to achieve optimal performance (in production environments you may not even have permission to do so). Dongting is completely modular (while still being compatible with java8), and the server can be deployed with a minimized JRE using jlink. Try it Build First build, the artifacts are under target/dongting-dist: If you want to install the artifact: The directory structure after build is as follows: Run server In the bin directory, run the following command to start the server: The server will start and listen on port 9331(servers internal communication, e.g., raft replication) and 9332(service port). By default, a DtKV instance with groupId 0 will be started. Run the following command to stop the server: Run benchmark Run the following command to start the benchmark client: You may need to adjust parameters to achieve maximum throughput, for example: Try Java 21 virtual threads (need Java 21) See my results (AMD 9700X 6C12T, ZhiTai TiPro 9000 running in PCI-E 4.0 mode): Client usage The built-in DtKV in Dongting supports the following features: • get/batchGet • put/batchPut • remove/batchRemove • list • mkdir • compareAndSet • putTemp/makeTempDir (node with ttl) • createLock/createAutoRenewalLock uses as a separator for keys. For example, you can access under the directory using . The value cannot be null or empty string. To use the client, you need to include the (300+KB), with no other dependencies. The following is a simple example of initializing the client: Make sure to specify the correct port. Each raft node exposes two ports: One is the **replicate port**, default 9331, which is used for internal communication such as replication between raft nodes. The also connects to this port. The other is the **service port**, default 9332, which is used for connections from clients like . provides synchronous and asynchronous interfaces. For a single , asynchronous operations achieve maximum throughput, while synchronous operations require many threads (or virtual threads) to reach higher throughput. It is important to note that callbacks for asynchronous operations may be executed on raft thread or IO threads. Therefore, you should never perform any blocking or CPU-intensive operations within these callbacks. If you are unsure or lack advanced skills, it is strongly recommended to use the synchronous interface. Here is a simple example of using the : For detailed usage of the class, please refer to Javadocs. Client examples • SimpleDemo - Basic put/get/list operations • BatchDemo - Batch read/write/delete • CasDemo - CAS atomic operations • TempValueDemo - Temporary value (TTL) • TempDirDemo - Temporary directory (TTL) • UpdateTtlDemo - Update TTL of temporary nodes • WatchDemo - Key/directory change monitoring • DistributedLockDemo - Distributed lock • AutoRenewalLockDemo - Auto-renewal distributed lock Server configuration Server configuration mainly consists of two configuration files: and . config.properties This file configures basic parameters of the Raft server: • **nodeId**: Each server must have a unique positive integer node ID, starting from 1. If there is only one node, i…