In my original post, I explained why we chose not to run our model on a server, but I should have clarified that the time and cost savings were specific to our project and are not generalizable to every model using a server approach. I updated the post to reflect this and to clarify our thinking behind this decision (updated 9/17/18).
At the Urban Institute, we create predictive models that a broad audience can use to estimate future trends or the potential effects of policy changes. Typically, we either calculate all potential outcomes beforehand and store them in the browser or run the model on a dedicated server. But those options weren’t practical for our latest version of the Prison Population Forecaster. Instead, we tried something new and ran the model directly in the browser.
In this post, I explain why we chose to run the Prison Population Forecaster in the browser and share the lessons we learned in doing so.
The Prison Population Forecaster is a data tool that allows users to explore how policy changes would affect state prison populations and correctional spending. Users can adjust the average length of prison stays and the number of new admissions to see population and cost forecasts over the next 10 years.
Running this kind of model directly in the browser can be tricky because the size of the data, combined with the model’s computational requirements, would typically result in slow browser performance. In this case, however, the dataset is small and the model is light so everything can be run in milliseconds. But moving the process to JavaScript — the interactive programming language of the web — was anything but smooth.
Our first thought: Let’s preprocess the data
When the first version of the Prison Population Forecaster was developed by Urban’s Justice Policy Center in 2015, the model results were stored in a single data file. When the user visited the tool, simulations were simply drawn from that existing data file. This worked well because there were a limited number of possible outcomes. The first forecaster allowed the user to select between 15 states or the federal government, 6 offense or admission types, 2 policy changes, and 4 reduction options for 768 possible results.
For the updated release, however, we planned for 18 discrete categories of offenses. For each category, users can increase or decrease length of stay and admissions in 1 percentage-point increments ranging from a decline of 100 percent to an increase of 100 percent, resulting in 201 potential choices (including zero, or no change, as an option). Plus, unlike in the previous version, the user can combine reductions across different offense types. These calculations are repeated across the 45 states for which the research team had data and could accurately model. Therefore, to preprocess all the possible results and store them in a database, we would have had to make 201^(18*2), or 8*10^1082 (8 followed by 82 zeros), calculations for each state.
For obvious reasons, our previous strategy of precalculated model runs was impossible.
That won’t work: Let’s try running it on a server
Next, we evaluated whether we could run the model on a dedicated server. In this framework, every time a user created a custom prison policy forecast, we would send the chosen variables to a server, which would run the model in R and then return the results to the user in the browser. We have used this setup for other Urban Institute modeling features using more complex and proprietary codebases, such as the Tax Proposal Calculator.
But two issues can arise with running the model on a server and communicating with that server every time the user changes an input variable.
1) It can be time consuming. To process the model on a server, the browser must communicate with the server, wait for the server to run the model, and wait for the server to send back the results. This usually takes only a few milliseconds and is unnoticeable on most tools. But shaving off a few milliseconds for each run of the model adds up to noticeable savings when you run the model many times consecutively. Because we had a design that allowed for multiple variable changes to be made in rapid succession, we opted to run the model in the browser, where we could capitalize on those millisecond savings to create a lag-free experience.
2) There are additional costs. Running a dedicated server costs money. A single server might not cost much, but a spike in traffic might require additional servers to keep the response times below 10 seconds. We would have to test the speed of our calculations and determine the number of requests that one server could handle. Once we hit this limit, our infrastructure would need to quickly add additional servers or have a few servers waiting to handle potential traffic surges.
Preprocessing the model was out of the question, and running it on a server seemed like more of a hassle than it was worth.
Why not run it in the browser?
Because our previous two options were infeasible or impractical, we looked at translating the original R script into JavaScript. If we could run the model within the browser without having to use an external database or server, we would save time and money. But this approach presented a few key issues:
First, in JavaScript, nothing protects us from having our model taken, broken apart, and reused elsewhere.
Because of the way web browsers parse information, anyone with web development skills could easily extract the code and data used to run our model. Luckily, this model was intended, from the beginning, to be an open-source product. And though the data must be made available in the browser, it is already anonymized and generalized in a way that protects the identity of individuals. We were therefore fortunate that there were no privacy issues or intellectual property concerns to address.
Second, JavaScript is fast with arithmetic but gets bogged down with complex statistical functions.
Many R-based models make use of R’s built-in statistical functions. These functions rely on complex mathematical formulas and are cumbersome to reproduce in JavaScript. Luckily, the Prison Population Forecaster uses simple arithmetic. Historical averages for lengths of stay and yearly admissions for each offense category are projected forward using simple weighted averages, assuming recent years are likely to be predictive of the future. Changing the expected length of stay or admissions rate of an offense category changes the projected number of people from the baseline, without requiring any complicated statistical functions. The most complicated function is the use of JavaScript’s built-in tangent function. Therefore, the forecasting model would have the ability to run quickly in JavaScript.
Finally, JavaScript’s internal calculator creates miniscule rounding errors because of how it stores numbers, which can lead to large errors if left unchecked.
Translating the model from R to JavaScript appeared to be straightforward until we tested the model by reducing the length of stay of a single offense category by 100 percent.
Logically, a reduction of 100 percent should always mean that the number of people incarcerated for the specified offense will be zero. Instead, the model reported that the number of people incarcerated would be several billion! What happened?
It turns out that a single particular scenario — generated by JavaScript’s underlying rounding method and its general (but not exact) accuracy — created a weird rounding issue. The proportion of people remaining in prison from the previous year is calculated by the following equation:
Because of how length of stay was being calculated, JavaScript could sometimes give us a small negative number, like -0.00000001, instead of a true zero, the way R or Stata would have. Thus, with a supersmall negative number on the denominator of the right-hand side of this equation, we would get a superlarge negative number on the left-hand side. A little additional conditional coding that checked for negative values for the length of stay variable solved this issue, all because of the way JavaScript was handling some of the math.
By addressing these three concerns, we could translate the model to the browser. This allowed us to reduce the model’s response time to milliseconds. Now, the predominant computation load for the browser is displaying the results, not calculating them. For those interested, we posted the original R script here and its JavaScript adaptation here.
Modeling in the browser wins
The tremendous work done by our Urban research colleagues in creating such a simple yet effective model allowed us to translate the model into the browser and create a fluid interactive tool. As with any model, we could have used other approaches, but we felt that running the model in the browser was best for our needs. Even though JavaScript’s eccentricities presented us a major calculation challenge, the performance improvement made this choice a no-brainer.
