Everyone has an opinion. Whether it’s a spirited discussion in a tong or an ugly rant in the Facebook comments section, we find ways to voice our feelings and frustrations. In a democratic society, everyone’s opinion matters. And to understand people’s views in a methodological manner, we have partnered with The Asia Foundation (TAF) to design and implement robust public perception surveys since 2019. Together, we have conducted three nationally representative perception surveys in Bangladesh on various issues like political governance, democracy, development, and societal issues.
But how do we ensure that the questionnaires enable people to express their true feelings and opinions accurately? Even wording a question differently can significantly sway responses. Based on what we learned through these surveys about what works and what doesn’t, here are a few things researchers should look for (and look out for) when designing and conducting a perception survey. It is worth noting that most of these issues apply when designing and implementing any kind of quantitative survey.
“It’s okay to not know.”
Not everyone keeps up with the country’s current issues or political affairs, and many do not care about them at all. It is important to have a clear understanding of what you assume respondents must know. For example, they may not have any experience attaining government services, voting in elections, or talking to government personnel. Researchers often forget that not knowing is a valid response.
Interestingly, when we followed our enumerators in the field, we found that respondents felt slightly embarrassed saying that they didn’t know something. They also tended to give positive responses to those questions. For example, if you ask respondents about their satisfaction with a government office, those who have never visited it tend to just say, “It’s good.”
Therefore, it’s recommended that you ask if they know about something first or keep options like “don’t know” and “no response.” This allows respondents to provide their correct perception or skip a question if they feel uncomfortable answering it.
Somewhat likely to use the Likert scale?
Depending on what kind of response you are looking for, consider when using the Likert scale is a good idea and when a simple yes/no question is more appropriate. Furthermore, when applying a Likert scale, don’t just use a five-response scale because it’s common. Often, the middle response of the five becomes a “neither-this-nor-that” response and allows respondents to be vague, consciously or otherwise. Just think, is the middle “neutral” option really necessary? Or are you simply providing an easy option for those who want to remain neutral without putting too much thought into it?
What do these words mean?
Be careful with words you think are common and everybody understands. Let’s say you are asking about violence in the respondent’s locality. What kind of violence are we looking for exactly: a scuffle between friends, something that sheds blood, or a life-or-death brawl between groups? Words like ‘trust,’ ‘independence,’ ‘empowerment,’ etc. may mean different things to different people, even when translated into Bangla. There are also words that respondents may have never heard of, like climate change, democracy, accountability, etc. Ensure you are on the same page as your respondents regarding every single word of the questionnaire.
Sometimes, respondents are more used to colloquial terms of words. In this case, enumerators may have more experience finding the right words than researchers. For example, we found that different Bangla terms for ‘national election’ are used in various parts of the country, like “এমপি নির্বাচন,” “রাজার ভোট,” etc.
Leading questions lead to a disaster
When we are talking about a concept or a recent issue, it often helps to provide a few sentences of context or explanation before getting into the questions about them. However, sometimes context can lead respondents to a certain response. If you give positive examples of a recent issue, a respondent may be swayed to provide a positive response. As a simple example, let’s say you ask if someone is satisfied with the current development trajectory of their locality, and give a context by providing a few examples of changes in the past few years. Those examples will likely determine if the respondent provides a positive or negative response.
Interpret results as perceptions rather than facts
It is important to remember that what people think and the actual events are two different things. While interpreting the results, it is necessary to assert that the findings are perceptions rather than facts. Vaguely written findings could even lead to newspapers misinterpreting the findings and creating confusion.
Avoid double-barrel questions
It’s very common to ask a double-barrel question accidentally. Let’s say you ask, “Did you approach your local or national government representative recently?” These are essentially two questions wrapped in one. Consider what you really want to find from this question. Based on that, you could ask if respondents met their local representative in one question and national representative in another. Or, if you do not care what kind of government representative they approached, you could make it clear to the respondent.
Watch out for social desirability bias
Respondents often avoid giving controversial, conservative, or uncomfortable answers just to make themselves look good. It’s important that enumerators create a non-judgmental and safe environment and ensure that other people or family members are not eavesdropping on the answers, which may obstruct respondents from speaking their minds.
Take a break if the questionnaire is lengthy
Our experience tells us that a small, few-minute break can actually reduce respondents’ fatigue and help them refocus. Enumerators can also ask for a glass of water just to allow the respondents to get up and move around and maybe check if any cooking on the stove needs to be attended to.
A pilot will make your questionnaire fly
Take your questionnaire out for a spin—as in, run some pilot surveys. Observe how the questions can be better delivered and how they are being perceived and responded to. New and unexpected responses may also emerge.
Voicing people’s opinions is the cornerstone of a democratic society. Public polls and opinion surveys help us understand the nation’s pulse on important issues. These opinions are shared in the newspapers and voiced by activists for the policymakers to listen to. But no method is perfect, and quantitative surveys often answer the “what” and can’t answer the “why.” It is necessary to look beneath the results because very different underlying factors may drive the same survey results. Qualitative methods can prove valuable in explaining the reasons for the results, but that may not always be feasible. Meanwhile, we can always sharpen our tool and ensure we get the best out of our survey efforts.
