Slides

https://github.com/keypointt/reading/blob/master/spark/2017_spark_spring_SF_Databricks_structured_streaming.pdf

note

1. Metric processing overview

Tech stack:

• Kafka
• Parquet

---

2. Watermarking example

Watermark: Data newer than watermark may be late, but allowed to aggregate

• moving threshold of how late data is expected to be and when to drop old state
• Trails behind max seen event time
• Trailing gap is configurable

Reference:

• Watermark blog: https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html
• https://www.youtube.com/watch?v=8o-cyjMRJWg&feature=youtu.be