We start by sending a survey to locals in emerging markets, relying on survey outreach methodology developed at MIT’s Jameel Abdul Latif Poverty Action Lab (J-PAL), the leading development economics research unit in the world.
We reach locals through SMS and interactive voice recordings (IVR) to pose binary questions for them to make predictions against. Our use of SMS and IVR allow us to reach a broader swath of potential respondents, rather than being limited to in-person collection in cities or biasing by income through smartphone apps. Using the information available to them, local respondents make predictions. Through their predictions, we hope to incorporate on-the-ground perspective, along with the broader social networks through which information is passed informally.
In the next step of our process, all respondents are paid in airtime minutes in exchange for their participation. This is to encourage higher uptake on our survey, but includes a crucial performance component – correct predictors will be paid a bonus. This is to encourage respondents to provide their best possible answer, rather than simply answering as many prediction requests as possible. By using airtime, we will avoid the challenges of currency conversion and dependence on active mobile money systems. We then wait for the event horizon to pass, after which responses are scored and we isolate the correct predictions. Correct forecasters are then paid a bonus for their accurate responses.
With a set of results identifying the correct respondents, we then add the latest prediction set to our database of prior predictions. At this point, we introduce machine learning processes to identify which forecasters tend to be most accurate, as well as which forecasters are most rapidly increasing in accuracy. The advantages of machine learning in our process are three-fold. First, incorporating this step allows us to systematically sort out and overweight the most consistent predictors, whom we dub “super-forecasters.” Those with longer histories of success are overweighted to continuously improve our accuracy. Secondly, systematically identifying top forecasters allows us to dynamically compensate respondents over time, so that we can increasingly reward individuals if they continue to produce accurate results. Third, having a central repository for predictions from each country will allow our predictions and populations of super-forecasters to become increasingly better over time. Because the pattern recognition algorithms improve with more data, the more queries we run, the better we will be able to match specific populations to specific topical predictions.