Data Cleansing
Is the process of detecting and correcting (o removing) corrupt or inaccurate records from a record set, table or database and refers to identifying incomplete, incorrect, inaccurate or irrelevant parts of the data and then replacing, modifying, or deleting the dirty of coarse data.
Data Cleansing â improves â Data Quality
Note
Data Issue Prevention is far better than data issue detection and cleansing, as it is cheaper and more efficient to prevent issues than to try and find them and correct them later.
How is data cleansed
- Manually â not ideal for large dataset
- Data Wrangling Tools
- Batch processing through scripting
- Data quality firewall
Data Cleansing Techniques
This is just a list of techniques, for more details check Datamanagement wiki
- Abbreviation expansion
- Clustering
- Cross-checking with a validated data set
- Remove duplicates
- Data enhancement
- Data harmonization
- Remove inconsistency
- Remove irrelevant data
- Merging
- Drop or impute missing value
- Remove outliers
- Parsing
- Patterns
- Removing typographical errors
- Standardization
- Statistical methods
- Transformation
- Type conversion
- Edit rules