MLOps Systems

MLOps is the system around the model: data, training, registry, release, serving, monitoring, incident response, governance, and cost control. The model artifact is only one piece.

System Components

Component Job
Data catalog Tracks sources, schemas, ownership, and classification.
Feature store Keeps offline and online features consistent.
Training pipeline Reproducible training and evaluation workflow.
Model registry Stores approved artifacts, metadata, metrics, and lineage.
Serving platform Routes traffic to model/runtime versions.
Monitor Watches data, behavior, latency, safety, and cost.
Release controller Handles canary, rollback, approval, and audit.

Deployment Modes

Mode Use Risk
Batch inference Offline scoring and reports. Stale predictions and backfill errors.
Online inference Low-latency request/response. Tail latency and feature freshness.
Streaming inference Continuous event decisions. Ordering, duplication, exactly-once expectations.
Shadow evaluation Test candidate without user-visible output. Privacy and extra cost.
Canary Limited production traffic. Requires per-version monitoring.

Practical Lab: Model Registry Record

model_id: ticket-escalation-v4
artifact_uri: s3://models/ticket-escalation/v4
training_data: dataset_2026_05_01
feature_schema: features_v7
eval_report: eval_2026_05_02
owner: ml-platform
approved_for: production-canary
rollback_to: ticket-escalation-v3

Cost Governance

Track cost by model, tenant, route, prompt tokens, output tokens, GPU hours, retrieval calls, reranker calls, and human review. Cost surprises are operational incidents.

Study Cards

Question

What does a model registry store besides weights?

Answer

Metadata, lineage, metrics, approvals, artifact URI, versions, owners, and deployment status.

Question

Why use shadow evaluation?

Answer

It tests a candidate on real traffic without making its output user-visible.

Question

Why is cost governance part of MLOps?

Answer

Token, GPU, retrieval, and review costs can regress independently of quality.

References