Shopify / maintenance_tasks

Maintenance Tasks A Rails engine for queuing and managing maintenance tasks. By ”maintenance task”, this project means a data migration, i.e. code that changes data in the database, often to support schema migrations. For example, in order to introduce a new column, it has to be added as nullable first, backfilled with values, before finally being changed to . This engine helps with the second part of this process, backfilling. Maintenance tasks are collection-based tasks, usually using Active Record, that update the data in your database. They can be paused or interrupted. Maintenance tasks can operate in batches and use throttling to control the load on your database. Maintenance tasks aren't meant to happen on a regular basis. They're used as needed, or as one-offs. Normally maintenance tasks are ephemeral, so they are used briefly and then deleted. The Rails engine has a web-based UI for listing maintenance tasks, seeing their status, and starting, pausing and restarting them. Should I Use Maintenance Tasks? Maintenance tasks have a limited, specific job UI. While the engine can be used to provide a user interface for other data changes, such as data changes for support requests, we recommend you use regular application code for those use cases instead. These inevitably require more flexibility than this engine will be able to provide. If your task shouldn't run as an Active Job, it probably isn't a good match for this gem. If your task doesn't need to run in the background, consider a runner script instead. If your task doesn't need to be interruptible, consider a normal Active Job. Maintenance tasks can be interrupted between iterations. If your task isn't collection-based (no CSV file or database table) or has very large batches, it will get limited benefit from throttling (pausing between iterations) or interrupting. This might be fine, or the added complexity of maintenance Tasks over normal Active Jobs may not be worthwhile. If your task updates your database schema instead of data, use a migration instead of a maintenance task. If your task happens regularly, consider Active Jobs with a scheduler or cron, [job-iteration jobs][job-iteration] and/or [custom rails_admin UIs][rails-admin-engines] instead of the Maintenance Tasks gem. Maintenance tasks should be ephemeral, to suit their intentionally limited UI. They should not repeat. [job-iteration]: https://github.com/shopify/job-iteration To create seed data for a new application, use the provided Rails file instead. If your application can't handle a half-completed migration, maintenance tasks are probably the wrong tool. Remember that maintenance tasks are intentionally pausable and can be cancelled halfway. [rails-admin-engines]: https://www.ruby-toolbox.com/categories/rails_admin_interfaces Installation To install the gem and run the install generator, execute: The generator creates and runs a migration to add the necessary table to your database. It also mounts Maintenance Tasks in your . By default the web UI can be accessed in the new path. This gem uses the [Rails Error Reporter][rails-error-reporting] to report errors. If you are using a bug tracking service you may want to subscribe to the reporter. See Reporting Errors for more information. [rails-error-reporting]: https://guides.rubyonrails.org/error_reporting.html Active Job Dependency The Maintenance Tasks framework relies on Active Job behind the scenes to run Tasks. The default queuing backend for Active Job is [asynchronous][async-adapter]. It is **strongly recommended** to change this to a persistent backend so that Task progress is not lost during code or infrastructure changes. For more information on configuring a queuing backend, take a look at the [Active Job documentation][active-job-docs]. [async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html [active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend Action Controller & Action View Dependency The Maintenance Tasks framework relies on Action Controller and Action View to render the UI. If you're using Rails in API-only mode, see Using Maintenance Tasks in API-only applications. Autoloading The Maintenance Tasks framework does not support autoloading in mode. Please ensure your application is using [Zeitwerk][] to load your code. For more information, please consult the [Rails guides on autoloading and reloading constants][autoloading]. [Zeitwerk]: https://github.com/fxn/zeitwerk [autoloading]: https://guides.rubyonrails.org/autoloading_and_reloading_constants.html Usage The typical Maintenance Tasks workflow is as follows: • Generate a class describing the Task and the work to be done. • Run the Task • either by using the included web UI, • or by using the command line, • or by using Ruby. • Monitor the Task • either by using the included web UI, • or by manually checking your task’s run’s status in your database. • Optionally, delete the Task code if you no longer need it. Creating a Task A generator is provided to create tasks. Generate a new task by running: This creates the task file . The generated task is a subclass of that implements: • : return an Active Record Relation or an Array to be iterated over. • : do the work of your maintenance task on a single record Optionally, tasks can also implement a custom method, defining the number of elements that will be iterated over. Your task’s will be calculated automatically based on the collection size, but this value may be overridden if desired using the method (this might be done, for example, to avoid the query that would be produced to determine the size of your collection). Example: Customizing the Batch Size When processing records from an Active Record Relation, records are fetched in batches internally, and then each record is passed to the method. Maintenance Tasks will query the database to fetch records in batches of 100 by defa…