Reading notes from 2020; revised November 17, 2026. Reported architecture and scale refer to the source period.
Treat the model as part of a service
The talk prompted me to organize production questions into data, prediction behavior, operations, and ownership. A model accuracy number cannot answer all four.
| Area | Question | Example evidence |
|---|---|---|
| Data | How old can features be? | Freshness distribution and missing-value policy |
| Prediction | What error matters to users? | A held-out metric and segmented analysis |
| Service | What latency and traffic must be supported? | Load-test distributions, not only averages |
| Recovery | What can be rolled back? | Model/data versions and a rehearsed procedure |
| Business | What outcome justifies operation? | A defined success measure and baseline |
| Team | Who responds when it fails? | Named ownership and a diagnosis path |
Separate data drift from service failure
An endpoint can be healthy while the input distribution changes. A model can score well offline while its online features arrive too late. Store enough identifiers and timestamps to investigate the relationship without logging unnecessary raw user data.
Make collaboration explicit
Scientists and engineers benefit from shared configuration, reproducible runs, code review, and agreed interfaces. A fixed staffing ratio does not follow from the presence of a data scientist; the needed support depends on experience, automation, and workload.
I prefer a small end-to-end exercise before a large launch: ingest representative data, train a candidate, serve a request, record a delayed label, and practice reverting it. That exposes handoffs that a notebook-only demo can miss.
Related: Michelangelo and ML tooling responsibilities.