erigontech / erigon

CI Guidelines Workflow triggers Each workflow should serve a distinct purpose with a distinct trigger pattern. Avoid duplicating workflows that fire on the same events — merge them instead. **Trigger ladder** (fastest feedback to most comprehensive): | Stage | Trigger | Suitable for | |---|---|---| | All commits | to / , on / | Lint, build, fast unit tests | | Merge queue | | Full test suite, race tests, integration tests | | Scheduled | | Flaky-test discovery, regression detection, QA | Run tests that developers can reproduce locally as late as possible — ideally only in the merge queue. This avoids burning runner time on every push and concentrates deterministic blocking checks at the point where they matter most. All workflows should include so they can be triggered manually without code changes. For workflows with inputs (e.g. QA), use dispatch inputs instead of separate workflow files. Draft PRs If a job uses to skip on draft PRs, the workflow **must** include in its event types: Without , converting a draft PR to ready-for-review fires an event the workflow doesn't subscribe to, so the job never runs — the PR appears to have skipped CI until the next push. Required checks and path filters Required checks must always report a status or they block the PR indefinitely. Do **not** use workflow-level / on workflows with required checks. Instead, use a step-level filter (e.g. ) so the job always runs and can report "skipped" rather than going missing. Runtime goals | Run type | Target | |---|---| | Cached (source unchanged) | < 5 minutes | | Cold / no cache | ≤ 30 minutes | If a job regularly exceeds 30 minutes cold, fix the underlying cause — don't just raise the timeout: • Split large packages into smaller ones so Go's test cache works at finer granularity. • Break a single large job into parallel jobs. • Move slow tests to the merge queue so they don't block developer iteration. PRs that make tests significantly slower should include workflow changes to compensate. Go test caching Go's test cache keys each package result by the compiled test binary hash **plus** the /size of every file the test opens at runtime. A single file with a wrong mtime invalidates the cache for the entire package. Fixture mtimes **Main repo fixtures** — run over testdata paths so each file's mtime equals the commit that last modified it. This is deterministic and content-sensitive. **Submodule fixtures** — shallow clones ( ) don't have enough history for . Instead, set every file's mtime to the submodule's HEAD commit timestamp. All files in a submodule share the same mtime, which is stable as long as the pinned commit doesn't change. **Directory mtimes** — normalize all directories to a fixed epoch (e.g. ). Git doesn't track directory mtimes, so without this step they reflect checkout time, which varies between runs. Keeping packages small Go caches test results at the package level. A package with hundreds of test cases that reads many fixture files will miss the cache if *any* fixture changes. Splitting it into sub-packages (one per fixture set) means unrelated fixtures don't cause unnecessary re-runs. This is also why the package is broken into focused sub-packages rather than one monolithic package. Required checks Required checks exist to block *regressions*, not to enforce perfection. • **Don't add a required check for a test that is already flaky.** Quarantine flaky tests first (skip them, move them to a scheduled run, or fix them), then make the check required. • **Checks must not be time-sensitive.** A test that passed yesterday on the same code should pass today. Non-deterministic failures (timing, network, external state) must not be required. • **Required checks should run on or , never only on to .** Checking after merge is too late — it creates a broken that blocks everyone else. Flaky tests Flaky tests should be discovered on scheduled runs that repeat the test suite, not tolerated silently in required checks. When a flaky test is identified: • Skip or quarantine it so it no longer blocks PRs. • File a bug and fix it separately. • Re-enable it as a required check once it is stable. Memory- and disk-intensive tests Some test packages allocate large databases or hold many files open simultaneously. Running too many of them in parallel can exhaust RAM or IOPS and cause OOM kills or spurious timeouts. Use to limit the number of packages tested in parallel (default: ), and to limit concurrency *within* a single package (default: ): These flags can be passed via in the Makefile: Consider setting tighter defaults in the workflow matrix for jobs that are known to be memory- or disk-heavy, rather than working around pressure by adjusting unrelated constraints like timeouts or GC tuning. Local reproducibility Every CI job should have a local equivalent so developers can pre-check before pushing. If you're changing code in , you should be able to run the corresponding test group locally ( ) and get the same result as CI. CI workflows should use the same Makefile targets that developers use locally (e.g. , ) rather than inline shell commands. This keeps CI and local test runs as similar as possible, reducing "works locally but fails in CI" surprises. Tests that can be reproduced locally should preferably only block in the merge queue, not on every PR push — this avoids burning runner time on checks developers can run themselves. Debugging Re-run with debug logging: Or in the UI: "Re-run jobs" → enable "Debug logging". Raw logs include per-line timestamps useful for profiling slow steps: