Research Data Services

Good analysis requires good data. While no dataset will be perfect, it’s important to rigorously evaluate the quality of your data and do your best to account for shortcomings you find.

Determine fit with your research goals

Using a valid, appropriate measure of what you’re studying is key. A good dataset for your particular project needs to include variables that can measure what you’re trying to measure! 

To determine fit, ask yourself:

• What am I trying to measure?
• Can I measure it directly (e.g., income, vote share, disease prevalence) or is it more abstract, requiring a proxy variable (e.g., social cohesion, happiness, intelligence)?
  • If it is abstract, how well does this proxy measure the concept? Why?
• Are the variables in this dataset measured at the appropriate level of analysis (e.g., individual, community, country)?
• Does prior literature or theory support the use of this measure? Do other people use it?

Understand the data generating process

The data generating process (DGP) is the real world process that creates the data you observe. The way your data was generated, for example who collected it and why, bears on its quality. Clear and transparent documentation for your dataset is useful in understanding the DGP and can also be a signifier of data quality. 

Answering the following questions can help you better understand the DGP:

• Which individual(s) or institution(s) collected the data (e.g., government, NGO, survey firm)? How was data collection funded?
• Why was the data collected? 
• How was the data collected?
• Do I understand the data collection methodology (sampling plan, methods used to collect data, etc.) ? Is there anything unclear about how it was collected?
• Is there anything about the DGP that might raise concerns? For example, was it an online survey in a country with limited internet access? Could the purpose or funding or source of the data collection drive bias?

Examine coverage and bias

High quality data is unbiased and complete. 

Run through these questions to check your data for comprehensive coverage and lack of bias:

• Are there missing values in the dataset? Is missingness random or systematic?
• Are certain places, people, or time periods left out of the data?
• Did I try different ways of analyzing the data to see if my results change? For example, do alternative variable codings?
• Could the way the data was collected affect the results? What kinds of bias—reporting, survivor, algorithmic, etc.—are plausible given the research context this data was collected in? 

Document (and do your best to account for!) shortcomings

• It is exceedingly rare to find a dataset that perfectly captures what you’re trying to test and is totally unbiased and complete. As a researcher, it is your responsibility to understand where your data is imperfect and try to account for those imperfections in your analysis, or at the very least, transparently discuss the shortcomings of your data and how they might impact the results you’re presenting. 

Ask for help

If you have any questions about evaluating your data, please reach out to the Research Facilitation team at Dartmouth Libraries. Our team page and email can be found here.