---

Apache Kafka

• 📖 Apache Kafka - Introduction
• 📖 Confluent.io > Learn

See also:

• Data Science Technology - Lecture 22 - Pub-Sub model
• In software engineering, the design pattern Observer is also called Publish-Subscribe

Apache Kafka is an open-source distributed event streaming platform used to build real-time data pipelines and streaming applications. It works like a high-throughput, low-latency message broker that lets systems communicate with each other using publish/subscribe (pub-sub) or stream processing patterns.

Use Cases

• Capture and analyize data from IoT Sensors like logistics, automative industry, factories
• Process payments and financial transactions in real-time like stock exchanges, bank, insurences
• To monitor patients in hospital care and predict changes in condition to ensure timely treatment in emergencies
• Provide real time analytics for business logistics and scientific research at scale to organizations.

How Linkedin Uses Apache Kafka:

• Monitoring metrics to ensure service/application health;
• Traditional messaging - various applications still uses the pub/sub messaging: from the search to the content feed and relevance
• Analytics: data tracked from users are all centrally collected and pushed onto an Hadoop grid for analysis and daily report generation
• As Core Building block (distributed log) by other products like a big data warehouse solution called Pinot, and the Espress distributed database.

An interesting stuff from this blog post:

4. Auditing service : Events get generated in one LinkedIn data center. However, they typically get moved to a different data center for a lot of the offline processing. During this move, it is important for the consuming application (e.g. map reduce jobs) to understand when it has received all events that were generated in a particular time window, so that it can begin the offline processing. The audit service, which is also built on top of Kafka, is used to solve this problem.

For Map Reduce, see also the Hadoop Module “Map Reduce”.

Confluent, is a startup that basically provides Apache Kafka as a service to companies

Image taken from confluent.io

Main Concepts

Event

An event records a fact that happens i.e a message a user send to another user. It is also called record or message in the documentation

Conceptually an event has a key, value, timestamp and optional metadata headers. Here’s an example taken from the website:

• Event Key: “Alice”
• Event value: “Payment of $200 to Bob”
• Event Timestamp: “Jun. 25, 2020 at 2:06 p.m”

Producers and Consumers

Producers are client applications that publish (write) events to Kafka. Consumers are those that subscribe (i.e read) these events. They can also process events.

An important concept is the decoupling: producers and consumers are decoupled.

Topic

Events are stored in topics. To simplify: a topic is similar to a folder in a filesystem, and events are files in that folder.

Topics are always multi-producer and multi-subscriber. Events are not deleted after consumption. Instead, you define for how long these events are retained, through a per-topic configuration setting.

Buckets: partitioning of topics

Topics are partitioned, so they are split across buckets located on different Kafka Brokers. This allow clients to both read and write data from/to many brokers at the same time.

Events with the same key are written to the same partition. Kafka guarantees that any consumer of a given topic-partition will always read that partition’s events in the exactly the same order as they were written.

Note

It’s similar to a sequential consistency model for DSM, the difference here is that Kafka guarantees it for partition. But is also similar to Pipelined consistency model because messages are written and read in a pipelined fashion way, with delays between production and consumption.

Replication

Every topic can be replicated across geo-regions or datacenters (i.e nodes). In this way there are always multiple brokes that have a copy of the data just in case things go wrong. The replication is done at the level of topic-partitions.

Event Streaming

“Event streaming is the practice of capturing data in real-time from event sources like databases, sensors, mobile devices, cloud services, and software applications in the form of streams of events; storing these event streams durably for later retrieval; manipulating, processing, and reacting to the event streams in real-time as well as retrospectively; and routing the event streams to different destination technologies as needed.” - source

Input is read from one or more topics in order to generate output to one or more topics, effectively transforming the input streams to output streams.

Kafka provides a Kafka Streams API that does not run directly on a Kafka cluster, it’s more like a client that listen (subscribe) to a topic and in real time process what’s in it. This approach is different from prod/consum, because in prod/consum approach, the read of data happens in i’d say sycnhronus way i.e the consumer read after the producer has finished, so it reads everything. While with streams, the client can read some of the data that the producer written, or even all but in real time and also processing it in real time.

Kafka Streams API can transform Kafka real-time data input streams (topics) into output topics without the need for an external stream processing cluster.

Kafka APIs

• Admin API
• Produce API
• Consumer API
• Kafka Streams API
• Kafka Connect API

See Kafka Docs - API

Kafka exposes all its functionality over a language independent protocol, however only Java clients are mantained as part of the main Kafka projet. The rest are maintained by the open source community.