This project tracks the dataset endpoints available in the U.S. Census Bureau APIs. There are no publicly available changelogs or regular announcements of added or removed endpoints. This scraper aims to fill that gap.
scripts/update-metadata.R gets the Census Bureau's dataset metadata json using the R censusapi package listCensusApis() function. This function selectively keeps and formats important metadata fields and ignores generic fields.
The script then checks to see what type of change has occured and outputs information accordingly.
Data change types:
Major: datasets added and/or removedMinor: changes to key metadata fields, like dataset names or descriptions, but no dataset additions or removalsNone: no changes to key metadata fields
All scraper runs update the update time displayed on the web page. Results are logged in data/update-log.csv.
On major data changes, endoint changes are saved out to the web page src/. On both major and minor changes the full metadata json is saved to data/data.json for posterity.
A Github action runs the scraper, commits changes, and rebuilds the web page. It posts to Bluesky if there are any major data changes.
Disclaimer: This project uses the Census Bureau Data API but is not endorsed or certified by the Census Bureau.