OptimalBits / bull

The fastest, most reliable, Redis-based queue for Node. Carefully written for rock solid stability and atomicity. Sponsors · Features · UIs · Install · Quick Guide · Documentation Check the new Guide! 🚀 Sponsors 🚀 Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ. 📻 News and updates Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library. Follow me on Twitter for other important news and updates. 🛠 Tutorials You can find tutorials and news in this blog: https://blog.taskforce.sh/ --- Used by Bull is popular among large and small organizations, like the following ones: --- --- Official FrontEnd Supercharge your queues with a professional front end: • Get a complete overview of all your queues. • Inspect jobs, search, retry, or promote delayed jobs. • Metrics and statistics. • and many more features. Sign up at Taskforce.sh --- Bull Features • [x] Minimal CPU usage due to a polling-free design. • [x] Robust design based on Redis. • [x] Delayed jobs. • [x] Schedule and repeat jobs according to a cron specification. • [x] Rate limiter for jobs. • [x] Retries. • [x] Priority. • [x] Concurrency. • [x] Pause/resume—globally or locally. • [x] Multiple job types per queue. • [x] Threaded (sandboxed) processing functions. • [x] Automatic recovery from process crashes. And coming up on the roadmap... • [ ] Job completion acknowledgement (you can use the message queue pattern in the meantime). • [ ] Parent-child jobs relationships. --- UIs There are a few third-party UIs that you can use for monitoring: **BullMQ** • Taskforce **Bull v3** • Taskforce • bull-board • bull-repl • bull-monitor • Monitoro **Bull <= v2** • Matador • react-bull • Toureiro --- Monitoring & Alerting • With Prometheus Bull Queue Exporter --- Feature Comparison Since there are a few job queue solutions, here is a table comparing them: | Feature | BullMQ-Pro | BullMQ | Bull | Kue | Bee | Agenda | | :------------------------ | :-------------: | :-------------: | :-------------: | :---: | -------- | ------ | | Backend | redis | redis | redis | redis | redis | mongo | | Observables | ✓ | | | | | | | Group Rate Limit | ✓ | | | | | | | Group Support | ✓ | | | | | | | Batches Support | ✓ | | | | | | | Parent/Child Dependencies | ✓ | ✓ | | | | | | Priorities | ✓ | ✓ | ✓ | ✓ | | ✓ | | Concurrency | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | Delayed jobs | ✓ | ✓ | ✓ | ✓ | | ✓ | | Global events | ✓ | ✓ | ✓ | ✓ | | | | Rate Limiter | ✓ | ✓ | ✓ | | | | | Pause/Resume | ✓ | ✓ | ✓ | ✓ | | | | Sandboxed worker | ✓ | ✓ | ✓ | | | | | Repeatable jobs | ✓ | ✓ | ✓ | | | ✓ | | Atomic ops | ✓ | ✓ | ✓ | | ✓ | | | Persistence | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | | UI | ✓ | ✓ | ✓ | ✓ | | ✓ | | Optimized for | Jobs / Messages | Jobs / Messages | Jobs / Messages | Jobs | Messages | Jobs | Install or _**Requirements:** Bull requires a Redis version greater than or equal to ._ Typescript Definitions Definitions are currently maintained in the DefinitelyTyped repo. Contributing We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop. --- Quick Guide Basic Usage Using promises Alternatively, you can return promises instead of using the callback: Separate processes The process function can also be run in a separate process. This has several advantages: • The process is sandboxed so if it crashes it does not affect the worker. • You can run blocking code without affecting the queue (jobs will not stall). • Much better utilization of multi-core CPUs. • Less connections to redis. In order to use this feature just create a separate file with the processor: And define the processor like this: Repeated jobs A job can be added to a queue and processed repeatedly according to a cron specification: As a tip, check your expressions here to verify they are correct: cron expression generator Pause / Resume A queue can be paused and resumed globally (pass to pause processing for just this worker): Events A queue emits some useful events, for example... For more information on events, including the full list of events that are fired, check out the Events reference Queues performance Queues are cheap, so if you need many of them just create new ones with different names: However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result. Cluster support NOTE: From version 3.2.0 and above it is recommended to use threaded processors instead. Queues are robust and can be run in parallel in several threads or processes without any risk of hazards or queue corruption. Check this simple example using cluster to parallelize jobs across processes: --- Documentation For the full documentation, check out the reference and common patterns: • Guide — Your starting point for developing with Bull. • Reference — Reference document with all objects and methods available. • Patterns — a set of examples for common patterns. • License — the Bull license—it's MIT. If you see anything that could use more docs, please submit a pull request! --- Important Notes The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processi…