Apache Kafka & Streaming Distributed Messaging Training

Introduction to Big Data, Big Data and Streaming, Kafka Introduction, Kafka Features, Kafka vs AMQP, Kafka Features, Kafka Use Cases

Introduction

Introduction to Big Data

Big Data and Streaming

Kafka Introduction

Kafka Features

Kafka vs AMQP

Kafka Features

Kafka Use Cases

Understanding Big Data and Data Streaming Problem Use Cases

Introduction to Data Ingestion

Messaging System

Data Streaming

Batch Processing

Live Data Processing

Data Forwarding, Queuing problems

Publish/Subscribe patterns

Workflow Application

Hadoop and Map Reduce Integration

Connected, Disconnected Architecture

Persistence issues with large data processing

AMQP/MQTT to Kafka Bridge

Kafka Architecture

Elements of Kafka [Consumer, Producer, Broker, ZooKeeper, Cluster]

Kafka Architectures

Kafka Overall Messaging Architecture

Kafka Streaming Architecture

Kafka Consumer and Producer Architecture

Kafka Persistence Architecture

Kafka Clusters

Kafka Consumer Workers Group

Kafka Setup [Hands-on]

Java JDK

Kafka Setup

Zoo Keeper Configuration

Understanding build-in tools, Kafka server, test publisher, test consumer

Configuration Kafka Single Node Broker

Start Zoo Keeper

Start Kafka Server

Using Example consumer and publisher

Broker, Consumer, Producer Configuration

Zoo Keeper connection

Broker ID

Logs

Port

Consumer Configuration

Producer Configuration

Kafka Administrator Commands [Hands-on]

List all topics

List all brokers

Create Topics

Change Topics

Delete Topics

Topics& Partitions

Understanding Topics

Understanding Partitions

Producer [Hands-on]

Kafka Producer Introduction

Kafka Producer API

Writing First Producer

Kafka Producer Class

Build a Kafka Producer

Configure Producers

Kafka Topics, Partitions Configuration

Send messages to Kafka

Send messages Synchronously

Send Message Asynchronously and process the async acknowledgement

Producing Keyed and Non-Keyed Messages

Serializers, Key/Value Serializer, ProducerRecord

Serialize Using Apache Avro, JSON

Working with Multiple Brokers

Consumer [Hands-on]

Kafka Consumer Introduction

Consumers and Consumer Groups

Subscribe for Topics

Consumer Records

Achieving higher performance with Consumer group

De-serialize key and values

Consumer Groups and Partition Rebalance

Creating a Kafka Consumer

The Poll Loop

Configuring Consumers

Commits and Offsets

Rebalance Listeners

Consuming Records with Specific Offsets

Kafka Internal Architecture, Storages, Cluster, Partitions in Depth

Cluster Membership

The Controller

Replication workflow, Replication Internals

Request Processing

Physical Storage

Reliability

Partition allocation and number of partitions

Broker Configuration

Using Producers in a Reliable System

Using Consumers in a Reliable System

Validating System Reliability

Performance Tuning in Kafka

Day 2

Cluster Architecture and Workflow

Revising Cluster, Producer, Consumer, Broker, Zoo Keeper

Data Flow Between all actors in Kafka Systems

Creating Clusters with Multiple Brokers

Broker IDs and Master/Slave Brokers

Fail over handling in clusters

Understanding Leader Broker

Balancing leadership

Cluster Administration and Configuration [Hands-on]

Configure Cluster

Replication

High Availability

Multi-Cluster Architectures

Apache Kafka’s MirrorMaker

Topic Operations

Partition in Depth [Hands-on]

Topics and Partitions

Understanding partition numbers

Default partitions

Explicit Partitions

Under the hood partitions

Replications

Understanding Replications

Replication Factors and High Availability

Messages in Details [Hands-on]

Message Models

Message Serialize

JSON handling

Message IDs

Persistence [Hands-on]

Persisting Messages on File System

Offline Consumer and Subscription handling

Durability of Messages

Failover and Persistence

Kafka Streaming

Stream Processing

Stream-Processing Concepts

Stream-Processing Design Patterns

Kafka Streams by Example

Kafka Streams: Architecture Overview

Kafka Monitoring

Considerations When Building Data Pipelines

Metric Basics

Kafka Broker Metrics

Client Monitoring

Lag Monitoring

End-to-End Monitoring

Kafka Connect APIs

Kafka Connect API overview

Kafka Connect

When to Use Kafka Connect?

Kafka Connect Properties

Perform File source and sink using Kafka Connect

Kafka Connect vs Consumer/Producer model

Kafka Integrate with Apache Spark

Integrate Apache Spark with Kafka

Spark Streaming Jobs and Kafka Stream

Production

Deployment

Configuration, cleaner installation

Gopalakrishnan Subramani

Instructor

Mentor, Trainer, Consultant & Architect for IoT, Azure IoT & Cloud, Node, React,
Angular, Scala, Apache Kafka, Apache Spark with deep expertise in building Industrial Scada, IIoT Solutions, Web, Mobile and Backend Applications on premise and cloud infrastructure.

Mentor and Consultant for SCADA, HMI Device application using modern web technologies, excellency in IoT, Sensor, M2M connectivity, building connected, wired, wireless device applications, cloud enabling, data streaming, RESTFul Architecture, Progressive Web Applications using Hadoop, Spark, Kafka, AMPQ, MQTT, MQTT-SN, CoAP, JavaScript, MongoDB, MySQL, Cassandra & Java.