Automated Policy Data Extraction for ProGov21.org Using AI
Project collaborators
Joel Rogers, Gabriel Kahan, Diana McFarland, and Walker Kahn, High Road Strategy Center
Matt Schleifer, DoIT
Steve Wangen, Abe Megahed, and Jason Lo, DSI
Project start and end dates
June 2024 – September 2024
The DSI team were excellent collaborators. Their help with designing an interface capable of scraping data with LLM-assistance for our digital policy library was a huge service. Their team was responsive, creative, and thoughtful in managing the project and its development. I’d recommend their work to anyone looking to implement similar cutting-edge tools.
Project summary
This project aimed to streamline how policy documents are processed and shared through ProGov21.org, a fully searchable digital library for local governments. Adding content to this site was time-consuming, as each document had to be manually found, reviewed, and labeled. This project developed an automated data pipeline that uses AI to extract and organize information from policy-focused PDFs collected from various sources.
The process starts with a custom-built data crawler that finds and downloads relevant policy documents from partner and public data provider websites. This crawler runs as a scheduled job using cron, and it stores documents and metadata in a centralized storage system.
Next, the documents are processed through a labeling pipeline powered by a large language model. This step extracts structured metadata and content such as policy title, date, jurisdiction, issue area, and summary, and maps documents to a pre-defined schema. This allows the data to be integrated into the ProGov21.org frontend through a RESTful API, where a human review ensures the accuracy of the data. Once evaluated by a person, the site uses the API to provide the broader public with structured access to the labeled policy documents, allowing end users to search and browse by multiple facets (e.g., issue area, state, year).
This new system drastically reduces the manual workload while maintaining quality and enhancing public access to policy data. The primary beneficiaries are local policymakers and members of the public looking for clear, accessible information on local, state, and federal policies. As a result of this project, this information is now easier to find and use, helping elected officials and community members stay informed and engaged.