Data Science - Pipeline

• 📑 What’s Data Science Pipeline?
• 📑 How to Work Through a Problem Like a Data Scientist
• Data Science for business professionals. A practical guide for beginners

The data pipeline is the end to end process, which defines how the data flows within the enterprise system for all the data needs.

The data pipeline is a broader term that encompasses many processes in the transfer of data from one system to another, including security, reading, serialization, transfer, transformations, writing, and other steps.

OSEMN Process

It’s an acronym that stands for Obtain, Scrub, Explore, Model, Interpret. It’s a list of common task in the work of a data scientist.

• Obtain: collect data
• Scrub: Clean the data, remove duplicates, remove outliers and fill or remove sample with missing values. see Data cleansing
• Explore: Exploratory Data Analysis
• Model: Model Selection and Evaluation
• Intepret: The model (hopefully) provide some useful insights to the Data Scientist, these are analyized and presented to public or stakeholders.

ETL process

Another simpler variant is a called ETL: **Extract, Transform and Load process.

In a pipeline, there may be many ETL sub-process running to deliver the data applications needed to run and store them.

Data pipeline properties

Data pipeline is a broader process that ensures that the system is efficiently and securely able to transfer data when required by the processes.

Another property is that a data pipeline does not start and end but is capable of taking data from any system to any system. This feature of the data pipeline makes them multi-directional in nature.

The data pipelines can be further categorized into three types, based on what kind of data flow they handle:

• Batch processing: Batch processing pipelines are the most popular and used to handle large volumes of data moving in/out of warehouses for various purposes of reporting, processing, and storage. The key feature of batch processing is that it is usually scheduled for a fixed time and does not require to deal with real-time data.
• Stream processing: these pipelines are designed to handle real-time data. The latency between the point of data origination and next process is nearly zero. This type of pipelines is used for real-time analysis of data in time-critical applications.
• Cloud-native processing: the cloud infrastructure provides multiple options and configurations to design the pipelines, native to cloud storage, and distributed services. They can be real-time, quasi real-time, batch, and can be configured to be hybrid. All the tools and infrastructure are provided by the cloud provider and pipeline managed by them. AWS, Azure, and Google cloud are the leading provider of managed cloud native pipelines.

Simple example/exercise of a pipeline

Design a data pipeline to get data from data.gov.in website for the Current Daily Price of Various Commodities from Various Markets (Mandi) and store that data in a CSV file.

Designing a data science pipeline

Designing a data pipeline is a core responsibility of a data engineer. A data engineer knows where the data get generated, how that can be transferred, the memory, and bandwidth limitations of the network, security best practices, and infrastructure status.

The key considerations before designing the pipeline could be to answer some of the following questions:

1. Where is the data source?

2. What type of data is to be extracted from the source?

3. What transformation to be applied to the data in the pipeline?

4. What is the destination of data?

5. Does the destination produce some data which needs to be transported or stored?

6. What is the nature of the pipeline: batch or stream?

7. Who can trigger the pipeline, or will be scheduled?

ETL vs ELT

ETL process does the transformation before loading the data. That step requires a clear understanding of end use of data. However there may be a lot of cases from the same data, which we do not know right now. In such cases, it’s better to load the data into a persistent store and let the applications transform it as per the requirement.

In modern data pipelines, the transform step is delayed until the end use is defined and accessed by the application. This way, the data can be stored in raw form, and later as per requirements, it can be transformed. This new process is named the ELT process, where the order of data process is extracted, loaded, and transformed by the applications. The concept of Data Lake follows the ELT logic of data flow in enterprises.

Scheduling jobs and messaging queue

Scheduler, as the name suggests, is a process that can periodically execute the process, that is, jobs. This is an important concept in data pipelines as the data pipelines need to be triggered by events.

In many cases, the trigger of the data pipeline is not just a particular time but could be other events as well. For example, we may want to get commodity price data whenever it rains, or we may want data when a new data gets updated, and more. In all such cases, the trigger to run data pipeline can come asynchronously and must allow you to run the data pipeline job either in real-time or as per some queuing logic.

A messaging queue mechanism allows the queuing of all such triggers (messages) and runs the desired application. The working of the messaging queue is simple; there are clients that create messages, called producers, and deliver them to the queuing system. Another set of applications called consumers, connects to the queue and gets the message processed by desired/targeted applications.