How to Hire an LLM Ops Engineer

• aKeep using stages — they remain the primary mechanism
• bReplace stages with run tags like env=production
• cUse registered-model aliases: set_registered_model_alias('model', 'champion', version) and load via models:/model@championCorrect
• dRename the experiment to 'production' so serving can find it

Explanation: Stages (Staging/Production/Archived) are deprecated in favor of aliases, which support more than one environment pointer per model and cleaner automation. Run tags live on runs, not versions; experiment names have no registry semantics; and stages are being phased out, not recommended.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the mid-level experience tier.

• aEmbed read-only AWS keys in every notebook
• bMake the S3 bucket public within the corporate network
• cSwitch the artifact store back to the server's local disk
• dRun the tracking server with --serve-artifacts so artifact reads and writes are proxied through the server, which holds the cloud credentialsCorrect

Explanation: --serve-artifacts turns the tracking server into an artifact proxy: clients talk only to the server, which owns the S3 credentials. Embedding keys leaks credentials, a public bucket is a compliance problem, and local disk removes the durability and sharing that motivated S3.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the mid-level experience tier.

• aDeclared entry points, parameters, and a pinned environment (conda or Docker), so mlflow run . -P alpha=0.3 re-executes the project identicallyCorrect
• bAuto-generated documentation for the model
• cA CI pipeline definition that Jenkins can read
• dA push hook that uploads artifacts to the model registry

Explanation: MLproject declares how to run the code (entry points, params) and the environment to run it in, which is exactly what one-command reproduction needs. It is not documentation, not a CI system, and it has no registry push hook.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the mid-level experience tier.

• aConcatenate the three pickles into one file and log that
• bSubclass mlflow.pyfunc.PythonModel: load assets in load_context(), implement predict(), and pass extra files via the artifacts parameter of log_modelCorrect
• cLog the pieces as generic artifacts and hand-write unpickling code in every consumer
• dFork MLflow and add a custom flavor

Explanation: PythonModel exists for exactly this: load_context restores arbitrary artifacts once per process and predict implements the inference chain, giving you a first-class pyfunc model. Concatenated pickles are unloads-able by standard tools, raw artifacts push plumbing onto consumers, and forking MLflow is absurd for this.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the mid-level experience tier.

• aIt compresses the artifact so downloads are faster
• bIt encrypts the model payload for compliance
• cThe signature pins the input/output schema (column names, dtypes, optionality), so serving layers can validate payloads and catch schema drift up front instead of failing deep inside model codeCorrect
• dIt pre-warms the GPU kernels used by the model

Explanation: A signature declares the expected schema, letting MLflow serving validate requests and fail fast on missing columns or wrong dtypes — a leading real-world cause of production model incidents. It provides no compression, encryption, or hardware warm-up.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the mid-level experience tier.

• aRepoint the alias: set_registered_model_alias('fraud_detector', 'champion', previous_version) — anything loading models:/fraud_detector@champion picks up the old versionCorrect
• bDelete the broken model version from the registry
• cRetrain the previous model from its run and register it as new
• dRename the registered model so serving falls back

Explanation: Aliases are pointers, so rollback is moving the pointer back — instant, auditable, and artifact-preserving (note: serving layers that cache loaded models still need a reload/restart to honor it). Deleting the version destroys rollback history, retraining is slow and non-identical, and renaming breaks all consumers.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the senior experience tier.

• aOnly raise the server's request timeout and keep logging per-iteration
• bBatch with mlflow.log_metrics(dict, step=...) or log_batch, and log less frequently — the server also caps batch sizes and payload sizesCorrect
• cLog the values as params instead of metrics
• dDisable tracking during training and reconstruct metrics afterwards

Explanation: Per-call logging is an HTTP request each time; batch APIs amortize that and servers enforce per-batch metric counts and payload limits, so chunking plus lower frequency is the fix. Raising timeouts treats the symptom, params cannot hold series, and post-hoc reconstruction loses fidelity and lineage.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the senior experience tier.

• aA nightly cron job that redeploys whatever the newest version is
• bAn engineer manually edits Kubernetes manifests after each experiment
• cA CI/CD job (or webhook listener) triggered by the alias/model-version event that resolves models:/name@champion and rolls the serving layerCorrect
• dAn email to the on-call asking them to copy files to the server

Explanation: Registry events (alias set, version created) are the natural trigger: automation resolves the alias to a version and deploys that artifact, giving a human gate with an audit trail. Cron redeploys remove the gate, hand-edited manifests are unauditable toil, and email-and-copy is not automation.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the senior experience tier.

• aIt just works — MLflow merges the series automatically
• bMetrics are deduplicated by step so only one process wins cleanly
• cAutolog detects DDP and serializes logging for you
• dInterleaved, duplicated series and backend store contention — log only from global rank 0, or give each rank its own child run and aggregateCorrect

Explanation: MLflow has no cross-process coordination: four writers produce duplicated/interleaved steps and hammer the store. The standard pattern is rank-0-only logging (optionally with aggregated values) or per-rank child runs. There is no auto-merge, no step dedup, and autolog does not handle DDP topology.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the senior experience tier.

• aThe signature is doing its job — fix the upstream dtypes, or deliberately re-log the model with an updated signature if the contract genuinely changed; disabling validation hides driftCorrect
• bSchema enforcement is always wrong in production and should be turned off
• cThe model file is corrupted and must be re-registered
• dThe serving container needs more memory to parse integers

Explanation: Signature enforcement failing is early warning of contract drift — the healthy responses are fixing the producer or versioning the contract explicitly. Blanket disabling removes your best drift detector, the artifact is not corrupt, and memory is irrelevant to dtype validation.

Why it matters: Evaluates practical understanding of MLflow concepts and problem solving for LLM Ops Engineer at the senior experience tier.