Earlier this year, I assumed the lead role of maintaining and improving the data engineering operations undergirding the Center on Nonprofits and Philanthropy’s (CNP) National Center for Charitable Statistics (NCCS) data. For 30 years, these data have made insights on the nonprofit sector accessible to researchers, policymakers, journalists, practitioners, and more through its open datasets and policy-relevant tools, such as the What’s the Financial Risk of Nonprofits Losing Government Grants? data visualization.

This year, CNP established three priorities to streamline NCCS operations: more-timely data updates, better data versioning practices, and stronger documentation of data provenance. These tasks have become increasingly challenging as the volume, complexity, and interdependencies of data have grown over time.

To meet these goals, the NCCS team turned to agentic tools to more easily orchestrate a data system that spans multiple GitHub repositories (repos). Urban’s Tech and Data team has deployed this approach across multiple projects to achieve efficiencies while maintaining quality standards.

In this article, I explain how we used Claude Code (CC) with Fable (on low effort) and Opus (on high effort) to develop our agentic programming tool and highlight the lessons we learned about agentic programming as a general approach. In sum, we found that

• an effective harness allows developers to conduct iterative technical reviews of existing code and make concrete and durable improvements;
• agents can keep an architecture consistent by ensuring that modules (datasets, APIs, dashboards, etc.) are synchronized and updates do not violate research requirements and technical specifications; and
• an agentic approach can greatly extend the capacity of smaller data science teams, provided all decisions and requirements are clearly documented for both human and artificial intelligence reference.

An Overview of NCCS Data

NCCS primarily maintains three publicly accessible datasets to support research on the nonprofit sector.

The IRS Business Master File (BMF) lists all nonprofits granted tax-exempt status by the Internal Revenue Service (IRS). The IRS uploads new versions monthly that reflect the active organizations. The NCCS team downloads each release, cleans and standardizes some fields, assigns each nonprofit a latitude and longitude for spatial joins, and creates composite datasets such as the Unified BMF, which tracks active and inactive nonprofits and can be linked to 990 panel data.

The NCCS CORE series contains a few variables derived from the IRS’ Statistics of Income 990 extracts. These data track nonprofit activities and fiscal metrics over time. NCCS has maintained a panel stretching from 1989 to 2024, with pre-2012 data coming from digitized paper filings, making it the longest-running panel of nonprofit finances available to the public.

The NCCS E-file database is a standardized and tabular version of the IRS’ raw XML releases of electronically filed 990 forms, published in rolling batches. These e-filed data contain approximately 2,000 fields of a mix of data types. Public data releases started in mid-2011 and were made mandatory in the Taxpayer First Act, beginning July 1, 2019.

Challenges with the Previous NCCS Pipeline

To understand how NCCS could better serve the field, we held weekly working sessions with our users. These sessions uncovered a clear split between long-term and newer users.

Researchers who had used NCCS data for years were comfortable with the existing formats, which read as a natural continuation of legacy datasets. Variable names encoded 990 form location (e.g., F9_REV_TOT for Form 990 total revenue) and were derived from a specific year’s IRS forms, which locked the data dictionary to that year and offered no protection against IRS schema changes. While this worked well for seasoned researchers who knew the ins and outs of IRS data processing standards, beginners needed hands-on guidance to get started.

Datasets can contain hundreds of variables across millions of rows, with metadata columns and file naming conventions that require domain knowledge or assistance to understand. NCCS-specific taxonomies (e.g., NTEEV2, a revised internal version of the IRS’ National Taxonomy of Exempt Entities (NTEE) codes that classify nonprofits by issue area) were internal formats that had not yet been adopted sector-wide. During these sessions, we provided key technical assistance, but our core priority following session feedback became democratizing access to data.

Two structural issues complicated our ability to change the pipelines.

1. Several pipelines drew on artifacts from long-time external collaborators, including open-source efforts like the Nonprofit Open Data Collective. These collaborations enhanced our ability to produce datasets more aligned with the Open Science Framework, but the dependencies weren't always documented with the provenance needed for clean end-to-end reproduction.
2. Incremental growth across years of different contributors meant NCCS code was spread across roughly 20 repositories, with no shared architectural overview. Clear documentation and cross-repository coordination was necessary.

Although the three primary components of NCCS’s data infrastructure are highly inter-dependent, they were each maintained using distinct data pipelines. This approach produced a set of recurring engineering challenges.

First, pipelines weren’t idempotent (they edited input data in place, such that running the same operations multiple times yielded different results), and they lacked test suites and validation reports. Second, updates were ad hoc, so the datasets were often out of sync, which made linking datasets challenging. Third, for historic reasons, data were decomposed by nonprofit 501c type and form type, making it hard for users outside of the nonprofit research community to understand which datasets to use for different use cases. Finally, downstream tools dependent on these outputs would sometimes pull stale data or hit broken links when upstream paths changed.

These challenges meant that our dataset updates were often out of sync with our public dashboard updates. In retrospect, building front-end tools while the underlying data infrastructure was still evolving created avoidable issues. Our user feedback sessions ultimately clarified the hierarchy of problems and crystallized a single overarching question: How could we update every component, keep everything synchronized, document the workflow, and streamline the system without breaking it?

How We Improved the NCCS Pipeline with Agentic AI Tools

We understood the opportunities for growth, but most data engineering solutions we found were designed for business use cases that require large, specialized teams and expensive licenses.

Instead, we sought to make our desired improvements in-house. By taking an agentic approach with Claude Code’s help, our actual costs came in well under budget and we saved time by treating the model as a technical collaborator. Drawing on our prior work with the Fair Housing Data Tool, we knew to work iteratively and in tandem with agentic tools.

We developed prompts that could be thought of as pseudocode, which abstracted syntax specifics and kept the developer focused on the “what” and the “why.” Then, we reviewed the output, course corrected the agent as needed, and produced additional iterations until we were satisfied, while keeping in mind that prompting can feel productive without being productive. Finally, we documented our decisions and checked the agents’ work.

From this process, we were able to make the following concrete, durable improvements:

One especially noteworthy improvement we made: Every pipeline run now writes a manifest, a JSON file recording the git commit SHA (unique ID to a code snapshot), the input file hashes, row counts, and sha256 checksums (a 64-character code used to verify file integrity and authenticity), so any dataset can be traced back to the precise GitHub commit that produced it. This change in process is the difference between “we updated the BMF in July” and “this row came from BMF release 2024–07, processed by commit id: 3f9c2d, with these data-quality checks passing.” That traceability allows us to use agentic tools with more confidence because we only trust agentic enforcement of a data contract if every artifact can prove where it came from.

Our final architecture is illustrated below:

1. The NCCS team sets the architectural intent, writes a document with an architectural decision record (called an ADR) and guardrails, and steers Claude Code sessions.
2. Each session proposes changes via pull request. The NCCS team reviews and approves the pull requests.
3. Each data pipeline generates the core data products, creating versioned outputs with a standard manifest (a unique ID containing schema, sha256, and row count).
4. The contracts repo is the single source of truth, holding the ADRs and the contract YAMLs (schemas and guarantees for where data should be stored and what datasets should contain) that every other layer reads.
5. A GitHub Action runs on every change, checking published outputs against their contracts. If a change goes against a documented decision or data contract, an issue opens automatically and a GitHub copilot agent investigates the issue and proposes a
6. Downstream packages, APIs, and dashboards each compose their own joins from the published datasets for specific use-cases.

A sample ADR looks like this (illustrative example, actual ADRs are on nccs-contracts):

```

# 0037 — Unified BMF: Versioned Builds with a 90-Day Deprecation Window

- **Status:** Accepted

- **Date:** 2026-06-29

- **Related:** [0014-standardize-manifest-shape], [0033-deprecation-window-policy]

## Context

The Unified BMF was previously overwritten in place with each new build.

This meant a researcher citing a specific version of the file had no way to

retrieve exactly what they downloaded, and a build with an undetected error

could silently replace the previous good copy with no fallback.

## Decision

Each build is published under a dated vintage folder and mirrored to a

/latest path. Previous builds are retained in the archive, reachable and

citable. When a new build supersedes an old one, both remain available for

90 days before the old version moves to the archive. This 90-day window is

a standing rule for any future supersession.

## Consequences

- Every published version is permanently retrievable and citable.

- A bad build can be rolled back without data loss.

- Users pulling /latest always get the current version with no change to

  their code.

- The 90-day window guarantees no one's workflow breaks without notice.

```

A single architecture repository now presents a consistent record, authored by the NCCS team, that every agent consults when evaluating pull requests. This record contains the data contracts (paths, schemas, and configs, all hardcoded), the intended system design, and a log of the decisions we've made, captured as ADRs.

When a decision is reversed, a new ADR supersedes the old one, links to it, and states why. For example, we initially decided that NCCS’s BMF and CORE should be merged into a single dataset. We reversed that decision after realizing it would be more efficient to query and join two smaller tables. The purpose of these ADRs isn't just to preserve project history and context, but to keep a consistent logic across iterations so no future decision quietly undoes the reasoning behind a past one.

Keeping the maintenance cost of this system low was a priority for us, so the agents aren't the first line of defense against drift (published data silently diverging from what its contract promises). We’re rolling out a GitHub Continuous Integration Action that runs before an agent is deployed: a cheap regex check (a fast text-pattern match) verifies that the paths in the contracts stay consistent, with all pull requests formatted to be legible to large language models. They contain keywords in the title and structured section headers.

If that guard fails, a second Action triggers, comparing the freshly published S3 (Amazon’s Simple Storage Service) artifact against its contract with its accompanying schema, manifest, checksums, and row counts. Any mismatch opens a [drift] issue that hands an agent exactly what it needs to work on: the contract, the relevant ADR, and the precise differences.

We do not require the use of Fable or Opus for our Copilot agents since guidelines are explicitly stated, and the agents do little in the way of interpretation or improvisation. Model intelligence is thus less important than execution, allowing us to keep costs meaningfully lower with cheaper models. We stick to Sonnet, corroborated by benchmark evaluations that have found similar performance between Sonnet and Opus.

Because the contracts repo is also the upstream source of truth that downstream repos draw on for context, the system can prevent data products from being out-of-sync with existing datasets.

This architecture repository is public, and you can read the contracts and ADRs here.

Impacts

Our efforts to optimize with agentic methods have begun to pay off. An analysis of downloads by unique IP addresses, with filters to reduce miscounts of bots, reveals that usage has accelerated:

2024 average: ~2,700 downloads per month from ~800 unique users

2025 average: ~4,900 per month from ~1,700 users

Early 2026 (January–April): ~8,000 per month from ~3,300 users

Summer 2026 (May–July): ~23,000 per month from 5,500–7,000 users, with August on pace to have ~33,000 downloads

Agentic methods are not solely responsible for this momentum. Growth was already underway, and agents equipped with early implementations, plans, and ideas from 2024 and 2025 were able to both accelerate improvements and improve the quality of implementation. The effect is visible in the data: After I began using Claude Code in March 2026, downloads tripled within two months.

Concluding Thoughts

We deliberately stopped short of turning agents loose at the coding level to build a system among themselves. The volume of output could easily overwhelm even a strong programmer at review time, and reading unfamiliar code is more taxing than writing it.

Instead, we ran several sessions concurrently, each tied to a part of the architecture we understood deeply. That let us drive the whole system in sync, using each interaction to notice when something was drifting from the intended design and to decide whether to steer it back or update the contract, akin to a systems architect.

Agentic orchestration can help data engineering teams manage a multi-repo, multi-pipeline system without sacrificing test coverage, validation, or release cadence. NCCS now releases versioned, documented datasets on a regular monthly cycle with full provenance. For public data teams looking to extend their capacity, agentic approaches offer a scalable path forward.