Fault Tolerance in Distributed Stream Processing Engines
IndoSys 2026 Tutorial
Speakers
- Abhilash Jindal, IIT Delhi
- Satyam Jay, IIT Delhi
List of Topics
- Basics of stream processing
- The problem of faults
- Distributed checkpointing algorithm used in Apache Flink
- Stop-the-world vs asynchronous checkpointing
- Consistent vs inconsistent checkpoints
- Hands-on: Implement asynchronous, consistent, distributed checkpointing in a simple Python-based standalone system
Pre-requisites
- Have done basic courses in OS, computer networks, and parallel programming
- Comfortable with Python
Expected Outcomes
- Understand how distributed stream processing engines work
- Learn about fault tolerance algorithms
- Implement an asynchronous consistent checkpointing and a recovery procedure
Reference Material
- The tutorial will implement the main algorithm described in [VLDB 2017] State Management in Apache Flink: Consistent Stateful Distributed Stream Processing
https://dl.acm.org/doi/pdf/10.14778/3137765.3137777 - This tutorial will implement checkpointing and recovery's main algorithm on a simple Python-based standalone system. The tutorial will be based on this lab:
https://github.com/codenet/col733-cloud/tree/main/labs/lab2 - Consistent checkpointing is discussed in classic papers:
- [TOCS 1983] Distributed Snapshots: Determining Global States of Distributed Systems
https://dl.acm.org/doi/10.1145/214451.214456 - [JPDC 1993] Virtual Time and Global States of Distributed Systems
https://nylas.github.io/paper-reading-group/papers/Virtual_Time.pdf
- [TOCS 1983] Distributed Snapshots: Determining Global States of Distributed Systems
Additional Setup / Equipment Requirements
A laptop with functional Python shall suffice.
