The Reach NC Voices project of EducationNC has had a number of ‘firsts’ this year. It became one of the first organizations in North Carolina, and one of the first in the country, to embed a web-based chatbot into news content. We also led the field on the use of online news and research to collect thoughts in every North Carolina county. And we are among the first to visualize the statistical validity of a survey sample using census data and participant demographic data.
The Reach NC Demographic Weighting tool represents a tremendous breakthrough for our project.
Why a representative sample matters
When posing questions to a given community, especially online, our distribution strategies are essential. It can be difficult to build a diverse pool of respondents online for a range of reasons. The challenge may lead to results that do not truly represent the people of any given community.
We think every voice and every story matters. We believe any conversation about the future of our state must reflect as much of the diversity of our state as possible. We worked with our development team at PublicInput.com to create a tool to make sure our results are representative of our state.
Demographic data on specific communities is readily available from the US Census Bureau, but that data is difficult to use in the context of community engagement. How can you connect your data to the census data?
We created a deep integration of census data sets tying user location, age, race, and gender to our responses.
The end result is a tool for a clear picture of who is participating, and more importantly, who we have missed.
Community-Specific Demographic Weighting
Consider a situation where a set of questions in one community receives 70 votes from males and 30 votes from females. But local census data for the community shows males are only 45 percent of the population. Without demographic weighting, the results will clearly skew toward male opinions.
And now we ‘weight’ with our new tool…
Once we know that a group is misrepresented, the tool will weight the responses accordingly to reflect a sample that reflects the census data.
How Reach’s technology automates the process
For an example like gender, the math is fairly simple. Automation becomes more valuable when layering in age, education, race, and marital status – combined.
For each of the major census demographic factors, the platform connects county-level census data to tailor weighting to the community.
This county-level demographic data allows our team to quickly see where samples are weak, or overrepresented. From here, we can alter our outreach strategy to reach audiences we missed and build a more robust dataset.
In doing so, the Reach team collects and shares results that more closely represent the community and captures accurate insights into what people think, feel, hope and believe.
A real-world example: Trust in news
As part of our initial statewide outreach, we asked participants to share which news outlets they most trusted. In the aggregate, it appeared newspapers were the most trusted format for North Carolinians:
When we applied weighting to accurately account for educational attainment in North Carolina, newspapers drop from ‘most trusted’ among 42 percent to 16 percent.
In the education demographics below, both bachelors and graduate degrees were overrepresented in the original results. When we increase the weight of voices with just some college attendance or a high school degree to reflect the reality of educational attainment according to the census we see decreased trust in newspapers and television, and more individuals who place their trust in alternative news outlets and radio.
Here is the visualized educational demography of our original set of respondents versus what one would expect in North Carolina according to the census data:
We could then offer information about the strength of the results and provide suggestions for how to target individuals not heard from previously:
What comes next?
As we expand efforts to elevate the voices of more North Carolinians, we will increase engagement in specific communities across the state this fall. By using localized demographic data combined with these powerful new tools, we will offer a more complete understanding of our communities.
Recently, the News Integrity Initiative announced a grant to support the next phase of our work. We have bold ambitions for the grant which will support a new set of tools that we will be building alongside our community-focused work. The Reach NC Demographic Weighting tool will be a critical piece of the next wave of our work — as will your voice.
Join us?
