An always-on Auto-ML service that continuously re-evaluates several candidate models on a real-time data stream and switches to whichever is currently performing best — without human intervention, even as the underlying data distribution changes over time.
The problem & constraints
Static models degrade silently as data drifts: a model trained once on historical data slowly becomes less accurate as real-world patterns shift (concept drift) or vary by season. Manually retraining or swapping models on a schedule is both slow and blind to when a change actually happened. The goal was a service that detects drift itself and re-optimizes without someone watching a dashboard.
Approach
- Every few minutes, the service re-evaluates a pool of candidate models (Hoeffding Tree, k-NN, SGD, Random Forest, among others) against the most recent window of the live stream.
- Two drift detectors — ADWIN and Page-Hinkley (via the
riverlibrary) — watch the error rate and flag a statistically significant change, which triggers re-evaluation outside the normal schedule. - The best-performing candidate at each check becomes the active model; the switch is logged so the history is auditable, not silent.
- A real-time dashboard (FastAPI + Plotly/Dash) surfaces the metrics that matter for trusting the system: rolling accuracy/F1, the history of model switches, inference latency, and CPU/RAM usage.
Trade-offs made
- ADWIN + Page-Hinkley over a single detector: ADWIN adapts its window size automatically but reacts more slowly to abrupt change; Page-Hinkley catches sudden shifts faster but is noisier on gradual drift. Running both covers more failure modes than either alone, at the cost of more false-positive re-evaluations to tune around.
- A fixed candidate pool over unbounded model search: unrestricted Auto-ML search space would slow down the re-evaluation cycle past the point of being "real-time." A curated pool of fast, well-understood online-learning-friendly models kept re-evaluation cheap enough to run every few minutes.
Results
The service maintained stable rolling accuracy/F1 across simulated drift and seasonality scenarios, with model switches visibly correlated to detector flags in the dashboard's history view rather than happening on an arbitrary schedule.
What I'd do differently
Formalize the re-evaluation interval as a tunable, drift-severity-aware parameter instead of a fixed period — reacting faster to severe drift and less eagerly to minor noise would reduce unnecessary model churn.