Data Quality
What is data quality?
Today, getting data on anything is easy, but not every data is inherently good. In many cases bad data can lead to false conclusions.
Data quality is the measure of how well data serves the purpose it’s intended for.
Data quality problems
Data quality problems can stem from incompleteness, inaccuracy, inconsistency, or data duplication. This happen when there are multiple copies of the same data, resulting in discrepancies.
Data with these issues must be cleansed
Data quality metrics
- Accuracy → measure whenever data is “correct” or not.
- Completeness → is all relevant data included?
- Consistency → is data the same wherever we look at it? Data consistency refer to a standard format and data collection methodology that avoids conflicts by looking at it (like thirtyseven vs 37).
- Formatting
- Data-entry rules
- Data normalization
- Integrity → Does data stay the same over time?
- Uniqueness → Is Each data point collected only once? If the same customer fills out the survery over and over, they skew the data toward their age bracket.
- Validity → Does the data make sense? For example, if an age is “3#”, we can discard that data point because ”#” is not a number.
Each of these elements need to be checked and managed with planning, rules and metrics to ensure that data can be used properly without creating false view of situations it represents.
- Missing data can skew whole datasets toward the data that remains.
- Formatting issues can result in duplicated data points, or even parallel sets of data that devolve into conflicting conclusions.
- Multiple rounds of inconsistent data handling can completely change the conclusions you might draw from the data.
Tips for data quality management
- Data collection and sources: Consider what data you need for your goals. Decide what variables you’re analyizing, then ask the sources you have available and how they cover this area. Then, consider specifics of the sources, such as their formatting and what data they may lacking.
- Data validation and monitoring: consider if data conforms to the expectations and standards set for that data. Ensure that all entries are formatted the same and have no duplicates in different formats. Consider if data points make sense for the format and data type.
- Timeliness: Spend some time considering when the data is needed, and what can be done to speed up and automate data quality management. This is expecially important for real-time data streams.