Obsidian + 🪴 Quartz 4.0

Outliers Removal in Python

2 min read

Outliar Removal in Python

Describe

Quote from Pandas Docs

DataFrame.describe(percentiles=None, include=None, exclude=None)

Generate descriptive statistics. Descriptive statistics include those that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values. Analyzes both numeric and object series, as well as DataFrame column sets of mixed data types. The output will vary depending on what is provided. Refer to the notes below for more detail.

Seaborn library

  • 📖 Seaborn documentation Seaborn is a library built on matplotlib used for displaying statistical graphics.

Some useful methods are listed here:

  • displot method: show the distribution of a data
  • boxplot method: “is a method for graphically demonstrating the locality, spread and skewness groups of numerical datas though their quartiles” (quote from Wikipedia)

Z-Score method in Python

t = training_dataset.copy()
t_den = t["density_readability"]
upper_limit = t_den.mean() + 3*t_den.std() # 3 standard deviation upper limit
lower_limit = t_den.mean() - 3*t_den.std() # 3 standard deviation lower limit
 
# Choose one of the following or try both and see who gives the best results
## Trimming
t = t[(t_den > lower_limit) & (t_den < upper_limit)]
 
 
## Capping the data
new_t = training_dataset.copy()
new_t.loc[new_t["density_readability"] > upper_limit, "density_readability"] = upper_limit 
new_t.loc[new_t["density_readability"] < lower_limit, "density_readability"] = lower_limit 
 
 
# Now you can visualize your data and/or train the model

Interquartile Range Method in Python

t = training_dataset.copy()
t_den = t["density_readability"]

Percentile Method in Python

Graph View

Backlinks