In 2019, a collaborative group of researchers, programmers, and designers set out to design an evidence-based tool that could support positive outcomes among Opportunity Zone (OZ) investments. As a result, the group launched the Opportunity Zone Community Impact Assessment (OZ Tool), which assessed the potential social impact of local development projects in OZs using Qualtrics survey instruments.
This initial tool launch was a beta test. With so many technological innovations and a desire to learn more about the types of projects assessed in the tool, the research team sought user feedback to plan improvements. As part of the launch, the team added a short user survey. Over the year-and-a-half-long beta testing period, from January 2020 to July 2021, the OZ Tool received more than 600 unique entries.
In July 2021, armed with this feedback and a new round of funding, the project team came up with three goals to pursue for the tool’s next phase: improve the tool’s ability to support more racially equitable outcomes, make the tool more user friendly, and improve its utility across a broader range of projects and places, including those outside of Opportunity Zones.
The resulting Capital for Communities Scorecard seeks to address these complex needs and to allow policy analysts and policymakers to better understand our society’s complex realities. We believe the development process and best practices we learned can inform other organizations looking to meet today’s policy challenges.
Testing the OZ Tool
The original tool included indicators that covered a range of social impact areas, including affordable housing, transportation and connectivity, and community wealth-building. To locally contextualize each score, the tool had users rank the six social impact scores by community members’ priorities. After standardizing scores across social impact areas and project types, the scores were multiplied by the weight attributed to the six social impact rankings.
To translate users’ Qualtrics entries into a project score and customized report, the Urban Institute’s tech and data team implemented a public-facing, cloud-based, serverless application stack using Qualtrics application programming interface (API) functionality with Python running on Amazon Web Services (AWS) Lambda to calculate a project’s score from the survey answers and generate a customized report.
The tool also piloted a method called “API as a service” and a novel licensing agreement (developed with the support of Urban’s legal team) that allows the project team to share the API with any external partner who wants to embed the tool on their website while Urban retains control of the intellectual property. This innovation had great potential; it meant Urban could create more evidence-based tools that could be seamlessly shared.
Creating the Capital for Communities Scorecard
As a first step in launching the new version of the tool, we broadened the scorecard questions to encompass projects outside OZs and consulted with racial equity experts at the Urban Institute and PolicyLink to reorient questions toward helping users achieve racial equity goals.
Next, we reworked the tool’s scoring mechanism to ensure answers reflected the question’s importance within its section. Although we kept the same overall scoring system as the previous tool, we made the scoring process more complex through the following:
· increasing the number of parent-child questions (when a person’s response triggers another question)
· embedding bonus points within answer choices
· expanding the harm scoring (when a project is penalized because of an answer that’s projected to hurt or disadvantage community members and their quality of life)
Because of the tool’s complex back-end scoring structure, the research team and Urban’s tech and data team worked closely to ensure projects were scored accurately. The research team tested projects in an Excel spreadsheet, where scores could be easily tallied, then passed the spreadsheet along to the tech and data team to see if they got the same score when running a project through the API. If the scores weren’t the same, the teams could go through the spreadsheet and API to see what wasn’t scoring properly.
On the usability side, the tech and data team successfully implemented the new features users requested:
· a new registration process that requires only their email address to get started
· a new table of contents so users can answer questions in any order
· a system that emails users a “retake link,” allowing them to use the tool again with all previous answers populated
We released the new Capital for Communities Scorecard on September 28, 2022, and it’s already garnered a lot of buzz from a range of private and public stakeholders, particularly federal and local equitable development public officials. The Capital for Communities Scorecard website also has had more than 2,400 page views since its launch.
Future use
Incorporating real data and feedback from a beta test into a rebooted final version is a potentially innovative model for other Urban tools used by practitioners and government officials. The ability to adapt the tool according to stakeholders’ feedback was crucial to achieving the tool’s goal and full potential and has led to a tool better suited for supporting projects that strengthen communities, boost racial equity, and benefit residents.
