kempnerforge.metrics.mlflow.models

The MLflow logged model of a checkpoint: one external model per run and step.

The training backend registers each committed checkpoint through this module, and the eval publisher finds or makes the model a checkpoint’s metadata.json names. Both pass a client bound to the tracking URI they mean, so the process-wide tracking URI is never read or changed. mlflow is imported only when a model is created or finished.

Functions

checkpoint_model_name(run_label, step)

The logged model name of the checkpoint saved at step by the run named run_label.

find_or_create_checkpoint_model(client, ...)

Id of the external model name of run run_id that every caller settles on.
kempnerforge.metrics.mlflow.models.checkpoint_model_name(run_label, step)[source]

The logged model name of the checkpoint saved at step by the run named run_label.

run_label is the run’s name, or its id when it has none. Characters MLflow rejects in a model name become _.

Parameters:
Return type:

str

kempnerforge.metrics.mlflow.models.find_or_create_checkpoint_model(client, run_id, name, step, checkpoint_dir)[source]

Id of the external model name of run run_id that every caller settles on.

client is an mlflow.MlflowClient; every call goes through it. That model is the earliest created (by creation time, then id) of the run’s models with that name that are typed checkpoint and READY or PENDING, read from every page of the search; a FAILED or UNSPECIFIED one, or one of another type (a model another tool logged under that name), is never used. So a checkpoint registered again (saved twice, re-run after a resume, or published) keeps one model, and callers that race to make it settle on the one made first. Nothing waits: a PENDING model, still being made by another caller or left by one that died, is finished here by uploading the MLmodel descriptor built from that model and finalizing it READY, only while it is not finalized READY or FAILED, so a caller that arrives after the model is finished does not write it again and a model marked FAILED meanwhile stays FAILED. A finalize that raises is ignored when the model reads READY afterwards. After finishing one the search is run again, so the search a caller settles on runs after its last write.

A model is made only when the search lists none to use, the way mlflow.create_external_model makes one: tagged external, typed checkpoint, with params step and checkpoint_dir, MLflow’s descriptor of an external model uploaded, then finalized READY; RuntimeError is raised if it was marked FAILED before that. The search is then run again, and the new model is returned only if it lists none to use. Server errors propagate.

MLflow has no atomic create-if-absent, so this narrows the race rather than closing it: a model that sorts earlier but becomes visible to searches only after another caller’s last search leaves the two on different models.

Parameters:
Return type:

str