swyxio / gh-action-data-scraping
this shows how to use github actions to do periodic data scraping
AI Architecture Analysis
This repository is indexed by RepoMind. By analyzing swyxio/gh-action-data-scraping in our AI interface, you can instantly generate complete architecture diagrams, visualize control flows, and perform automated security audits across the entire codebase.
Our Agentic Context Augmented Generation (Agentic CAG) engine loads full source files into context on-demand, avoiding the fragmentation of traditional RAG systems. Ask questions about the architecture, dependencies, or specific features to see it in action.
Embed this Badge
Showcase RepoMind's analysis directly in your repository's README.
Repository Overview (README excerpt)
Crawler viewgh-action-data-scraping this repo shows how to use github actions to do automated data scraping, with storage in git itself! **free git storage and scheduled updates!!!** 2021 Update You can read more in the Blog Writeup. As of May 2021, Flat Data scraping is officially supported by GitHub, check them out. Basic Idea • You set a cron triggered github action (cron examples - max frequency every 5 mins) • it checks out your repo with https://github.com/actions/checkout • and run your scrape script, write files to somewhere in your repo. This repo uses Node, but you can use whatever language you want • check it back in with https://github.com/mikeal/publish-to-github-action The script looks like: How it should look For people new to GH actions, this is how my Actions tab of this very repo looks if you need a reference point: Limits You can do whatever you like with this, including taking screenshots of sites! The limits I can think of are the limits of GitHub and GitHub Actions: • The max frequency of cronjobs on GitHub actions is every 5 minutes. For more frequent scraping, you will have to turn elsewhere. • GitHub has a soft storage limit of 1GB • You can work around this with Git LFS if you have to! • Actions are free for public repos, but incur costs for private repos • You get 6 Concurrent jobs, 1000 API requests an hour, and each job can take up to 6(!) hours In addition to these limits, GitHub Actions should not be used for: • Content or activity that is illegal or otherwise prohibited by their Terms of Service or Community Guidelines. • Cryptomining • Serverless computing • Activity that compromises GitHub users or GitHub services. • Any other activity unrelated to the production, testing, deployment, or publication of the software project associated with the repository where GitHub Actions are used. In other words, be cool, don’t use GitHub Actions in ways you know you shouldn’t. Be a good citizen, **don't abuse it and F this up for the rest of us**! This is heavily based on • https://github.com/mikeal/daily/blob/master/.github/workflows/daily.yml