Reading notes from 2020; revised November 17, 2026. Reported architecture and scale refer to the source period.

Treat the model as part of a service

The talk prompted me to organize production questions into data, prediction behavior, operations, and ownership. A model accuracy number cannot answer all four.

Area Question Example evidence
Data How old can features be? Freshness distribution and missing-value policy
Prediction What error matters to users? A held-out metric and segmented analysis
Service What latency and traffic must be supported? Load-test distributions, not only averages
Recovery What can be rolled back? Model/data versions and a rehearsed procedure
Business What outcome justifies operation? A defined success measure and baseline
Team Who responds when it fails? Named ownership and a diagnosis path

Separate data drift from service failure

An endpoint can be healthy while the input distribution changes. A model can score well offline while its online features arrive too late. Store enough identifiers and timestamps to investigate the relationship without logging unnecessary raw user data.

Make collaboration explicit

Scientists and engineers benefit from shared configuration, reproducible runs, code review, and agreed interfaces. A fixed staffing ratio does not follow from the presence of a data scientist; the needed support depends on experience, automation, and workload.

I prefer a small end-to-end exercise before a large launch: ingest representative data, train a candidate, serve a request, record a delayed label, and practice reverting it. That exposes handoffs that a notebook-only demo can miss.

Related: Michelangelo and ML tooling responsibilities.

References