.. _checkpoint-analysis: Analyzing saved checkpoints =========================== Analyzers can run on saved checkpoints without training again. The workflow has four steps, each with its own command: #. ``dendritic-analyze-checkpoint`` runs an analysis profile on one checkpoint; #. ``dendritic-submit-checkpoint-analysis`` runs it on many checkpoints as one Slurm job array, from a manifest; #. a collector checks every finished job and gathers its results into checkpoint-level tables; #. ``dendritic-summarize-checkpoint-metrics`` summarizes those tables across independently trained checkpoints, with bootstrap intervals and paired comparisons. The collectors keep technical repeats (support draws, noise draws, permutations) within each checkpoint and never pool trained checkpoints; inference across checkpoints happens only in the summary step. :ref:`cli-reference` lists every flag of these commands. This page also covers ``dendritic-select-validation-epochs``, the Poisson exposure-transfer commands and the analyses of recurrent models by task condition (:ref:`recurrent-condition-analyses`). .. contents:: On this page :local: :depth: 2 Running analyzers on one checkpoint ----------------------------------- .. code-block:: bash dendritic-analyze-checkpoint \ --config run_dir/config.json \ --checkpoint run_dir/final_model.pt \ --analysis-config analysis_profile.yaml \ --output-dir posthoc/run_0 ``--config`` is the resolved training configuration of the run (the ``config.json`` of its run directory, or an equivalent YAML file). ``--analysis-config`` is the analysis profile: a file with an ``analysis:`` root, or the analysis mapping itself, in the format of :doc:`analysis`. It replaces the training configuration's ``analysis`` section; the two are not merged. An unset ``analysis.runtime`` mode follows the training configuration's ``distributed.mode`` (see :ref:`analysis-runtime-profiles`). .. code-block:: yaml analysis: runtime: final: mode: stream batch_size: 512 performance: enabled: true final_splits: ["validation", "test"] noise_perturbation: enabled: true params: n_samples: 10 mse: false cosine_similarity: false multiplicative_gain: enabled: true params: gain_max: 4.0 n_steps: 5 accuracy: true The command rebuilds the datasets and the model from the training configuration with its recorded seeds, loads the checkpoint, and runs the profile's analyzers once with the result name ``--filename`` (default ``final``) on ``--device`` (default: the GPU when one is visible). It never trains or changes the checkpoint. The checkpoint may be a state dictionary or a mapping that holds one under ``model_state_dict``, ``state_dict`` or ``model``; a ``module.`` prefix on every key is removed. Loading is strict: missing or unexpected keys raise unless ``--no-strict`` is given, which records them instead (the collectors reject such runs). ``--output-dir`` becomes the run directory: the analyzer directories of :ref:`analysis-outputs` are written there, together with - ``checkpoint_analysis_provenance.json``: the paths and SHA-256 hashes of the configuration, checkpoint and analysis profile, the resolved seeds, the dataset sizes, the checkpoint-loading record (strict loading, missing and unexpected keys), the enabled analyzers, ``analysis_status`` (the record of the pass, with the status of every analyzer; see :ref:`analysis-status`) and the code identity; - ``analysis_status.json``, the same status record; - ``resolved_seeds.json``, ``runtime_environment.json``, ``train.log`` and ``dendritic_modeling.log``, as in a training run (see :ref:`run-outputs`). Dataset fingerprints are not written. The training-time rule that a sealed test split rejects every analyzer is not applied here; the size bound ``experiment.max_test_samples`` still is (see :ref:`dataset-splits`). Job files ~~~~~~~~~ Arrays of checkpoint analyses, and every collector, use a job file in JSON Lines format: one JSON object per line, one line per checkpoint, with the four arguments of ``dendritic-analyze-checkpoint``: .. code-block:: json {"config": "runs/seed_0/config.json", "checkpoint": "runs/seed_0/final_model.pt", "analysis_config": "profiles/noise.yaml", "output_dir": "posthoc/seed_0"} {"config": "runs/seed_1/config.json", "checkpoint": "runs/seed_1/final_model.pt", "analysis_config": "profiles/noise.yaml", "output_dir": "posthoc/seed_1"} Optional keys are ``filename`` and ``device``, and ``config_sha256``, ``checkpoint_sha256`` and ``analysis_config_sha256``, which some collectors use (see :ref:`checkpoint-collectors`). Collectors copy every key into their tables as a ``job_`` column, so further keys (for example ``seed`` or an arm label) become the design columns of the summary step. Relative paths resolve against the working directory of the command that reads the file, and every path must match the one recorded in the job's provenance. Running many checkpoints as a job array --------------------------------------- ``dendritic-submit-checkpoint-analysis --manifest manifest.yaml`` prepares the job files of a set of checkpoints and submits them as one Slurm array, one task per checkpoint. Each task runs ``python -m dendritic_modeling.scripts.analysis.run_checkpoint_analysis_job``, which checks the source checkout and then runs ``dendritic-analyze-checkpoint`` with strict loading. .. code-block:: json { "schema_version": "checkpoint_analysis_manifest_v1", "name": "noise-robustness", "output_root": "posthoc", "analysis_config": "profiles/noise.yaml", "completion_artifacts": ["noise_perturbation_analysis/final"], "jobs": [ {"id": "seed_0", "config": "runs/seed_0/config.json", "checkpoint": "runs/seed_0/final_model.pt", "seed": 0}, {"id": "seed_1", "config": "runs/seed_1/config.json", "checkpoint": "runs/seed_1/final_model.pt", "seed": 1} ], "pilot_job_ids": ["seed_0"], "scheduler": {"time": "02:00:00", "cpus_per_task": 4, "memory": "16G", "gpus": 1} } Manifest keys (paths are relative to the manifest file, including the paths inside a ``jobs_file``): ``schema_version`` ``checkpoint_analysis_manifest_v1``. ``output_root`` (required) Directory of the job outputs and of the ``jobs/`` ledger. ``analysis_config`` (required), ``analysis_config_sha256`` The analysis profile of every job, and an optional hash it must match. ``completion_artifacts`` (required) Relative paths, inside a job's output directory, that mark the job as complete together with ``checkpoint_analysis_provenance.json``. ``jobs`` or ``jobs_file`` (exactly one) The jobs, or a YAML file with ``schema_version: checkpoint_analysis_jobs_v1`` and a ``jobs`` list. Every job needs a unique ``id``, ``config`` and ``checkpoint``; ``output_dir`` defaults to ``/``; ``config_sha256`` and ``checkpoint_sha256`` must match when given. Other keys (for example ``seed``, ``filename`` or ``device``) are copied into the job file. ``repository``, ``required_commit``, ``require_clean_worktree`` Jobs run from a Git checkout of the library: ``repository`` (default: the checkout that contains the imported package) must hold the imported package under ``src/``. Its HEAD is recorded, must start with ``required_commit`` when that is given, and must match when each task starts; with ``require_clean_worktree`` (default ``true``) tracked files must be unmodified. ``pilot_job_ids`` The jobs that ``--pilot`` runs. ``name`` (default ``checkpoint-analysis``), ``scheduler`` The Slurm job name, and ``account``, ``partition``, ``time`` (``06:00:00``), ``cpus_per_task`` (8), ``memory`` (``32G``), ``gpus`` (0) and ``max_concurrent_jobs`` (a positive integer; default: all pending jobs). The command writes into ``/jobs/``: ``jobs_all.jsonl`` with every declared job (a later preparation with different content raises), ``jobs_