Runs and outputs ================ Every ``dendritic-train`` run writes its configuration, its records and its results into one run directory. This page describes where that directory is, every file a run writes, how the commands log, how they report failure, and which records let you repeat a run. :doc:`quickstart` shows a complete run; :ref:`cli-reference` lists every command-line option. .. _run-outputs: Run directory layout -------------------- Run directory name ~~~~~~~~~~~~~~~~~~ A run writes into .. code-block:: text /_/ where the timestamp is the local time at which the run starts. The defaults are ``outputs.results_dir: results`` and ``outputs.run_name: ei_sweep``, so a config without an ``outputs`` block writes to ``results/ei_sweep_/``. Set both in every config: .. code-block:: yaml outputs: results_dir: outputs run_name: population_network_feedforward ``dendritic-train --output_dir `` replaces ``outputs.results_dir`` and ``--run_name `` replaces ``outputs.run_name``. A relative ``results_dir`` is resolved against the current directory, and it is created when it does not exist. Two cases use ``results_dir`` itself as the run directory, without a timestamped subdirectory: * ``outputs.exact_run_dir: true``. Tools that claim an output directory in advance set it: the configurations that ``dendritic-sweep`` generates write to ``/results/config_`` this way (see :doc:`sweeps`), and ``dendritic-analyze-checkpoint`` uses it for its ``--output-dir``. * A ``results_dir`` whose last component is ``config_``, as in configurations generated by earlier sweep versions. A run never shares a timestamped directory: when another run with the same ``results_dir`` and ``run_name`` started in the same second, the later run writes to ``__1`` (then ``_2``, ...). Under ``torchrun``, rank 0 creates the run directory and writes the run's files; the other ranks receive the directory's path from rank 0. Files in a run directory ~~~~~~~~~~~~~~~~~~~~~~~~ This is the directory of a one-epoch CPU run of ``configs/examples/population_network_feedforward.yaml`` (see :doc:`quickstart`): .. code-block:: text population_network_feedforward_/ |-- analysis_status.json |-- config.json |-- dendritic_modeling.log |-- final_model.pt |-- init_gate_stats.json |-- main_network/ | `-- standard_best_model.pt |-- model_resources.json |-- nparams |-- operator_card.json |-- performance/ | `-- final.json |-- post_calibration_gate_stats.json |-- reactivation_calibration.json |-- resolved_seeds.json |-- runtime_environment.json |-- train.log `-- training_summary.json The files, in the order a run writes them: .. list-table:: :header-rows: 1 :widths: 30 70 * - File - Contents and when it is written * - ``train.log``, ``dendritic_modeling.log`` - The run's log files; see :ref:`logging`. * - ``resolved_seeds.json`` - Every random seed the run uses (``seed``, ``dataset_seed``, ``split_seed``, ``model_seed``, ``topology_seed``, ``loader_seed``, ``evaluation_seed``, ``probe_seed``), ``strict_deterministic``, and the numerics settings applied to PyTorch (``numerics``: float32 matmul precision, TF32, cuDNN determinism and benchmarking, deterministic algorithms). See :ref:`random-seeds`. * - ``runtime_environment.json`` - What ran: the imported package path, the checkout's commit, branch and a digest of uncommitted source changes, the Python and PyTorch builds (CUDA, cuDNN, threads, default dtype), the versions of NumPy, SciPy, scikit-learn, pandas, Triton, torchvision and OmegaConf, the visible devices, the process group, the thread, CUDA and ``DENDRITIC_*`` environment variables, the platform and the host name. * - ``dataset_fingerprints.json`` - Digests of the training, validation and test splits. Written only with ``experiment.record_dataset_fingerprints: true``. * - ``init_gate_stats.json`` - Statistics of every reactivation gate's parameters after the model is built, with warnings about suspicious values, and the initialization policy each dendritic layer applied to its gate (``init_policy``, counted under ``init_policies``). Written by every run; for a model without gates it lists no modules. * - ``config.json`` - The resolved configuration: every field of the typed schema with its default filled in, after the command-line overrides. It also records the seeds filled in for the analyzers, the gradient-clipping mode the strategy applies and the distributed mode the run trained with. ``dendritic-train`` and ``dendritic-analyze-checkpoint`` accept it in place of the original YAML file. * - ``reactivation_calibration.json``, ``post_calibration_gate_stats.json`` - The calibration of data-driven gates (a gate ``init_policy`` such as ``occupancy_quantile``; see :ref:`gate-initialization`) and the gate statistics after it. Written only when a gate uses a data-driven policy; a run that starts from ``model.initial_checkpoint`` skips the calibration. * - ``initial_checkpoint_provenance.json`` - Written when the run starts from ``model.initial_checkpoint`` (:ref:`initial-checkpoint`): the checkpoint's path and SHA-256 and the keys that did not match the model. * - ``main_network/`` - Checkpoints of the main training phase; see `Checkpoints`_. * - ``nparams`` - JSON (without a file extension) holding the total parameter count, ``{"nparams": }``. * - ``model_resources.json`` - Parameter counts (total, trainable, core, synapse candidates, active synapses), per-synapse-module records, the persistent recurrent state per sample, the device and, on CUDA, the peak memory. * - ``operator_card.json`` - The operator axes of the built core: weight domain and transform, integration rule, gate type and initialization policy, morphology (branch factors, leak conductance) and synapse mechanism, plus the result of the ``biological_neuron`` check (see :ref:`biological-constraints`). * - ``training_summary.json`` - The best epoch and its validation loss, the training and validation loss of every epoch (the training loss also split into its base and regularization terms), the data-loader settings and the data-parallel sizes (world size, batch size per rank, global batch size, steps per epoch). Model states are left out. * - ``final_model.pt`` - The state dictionary of the final model; see `Checkpoints`_. * - ``analysis_status.json`` - The completion status (``completed``, ``skipped``, ``failed`` or ``empty``) of every analyzer each analysis pass was asked to run; written by the passes that run at least one analyzer. See :ref:`analysis-status`. * - ``/`` - One directory per enabled analyzer, for example ``performance/`` with ``final.json`` (accuracy, AUC and log-likelihood on each split) and, with ``analysis.performance.training: true``, ``epochs/epoch.json`` and evolution plots. See :ref:`analysis-outputs`. ``dendritic-train --validate-only`` builds the datasets, the model, the calibrated gates, the analysis manager, the optimizer and the trainer, then stops. It writes the configuration and construction records above (``config.json``, the seed, environment and gate files, ``nparams``, ``model_resources.json``, ``operator_card.json``, the logs and an empty ``main_network/``) and no training results. ``training.main.strategy: presentation_stream`` also writes one line of metrics per epoch to ``main_network/metrics.jsonl`` (the file name is ``training.main.strategies.presentation_stream.metrics_file``). .. _run-checkpoints: Checkpoints ~~~~~~~~~~~ Checkpoint file names start with a prefix that names the training phase, for example ``standard_`` for ``training.main.strategy: standard``. ``main_network/best_model.pt`` The weights of the epoch with the lowest validation loss. ``training.main.common.save_best_checkpoint`` sets when it is written: ``every_improvement`` (default) after each epoch that improves the validation loss, ``end`` once after training, ``never`` not at all. ``main_network/checkpoints/epoch_.pt`` Written only with ``experiment.checkpointing.enabled: true``, every ``experiment.checkpointing.save_every_n_epochs`` epochs (default 10). ``epoch_0.pt`` holds the weights before the first epoch. These files hold model weights only, not the optimizer state. ``main_network/loss_curves.png`` The training and validation loss curves, written when ``training.main.common.plot_losses`` is true (default) and the run trains for more than one epoch. ``final_model.pt`` The weights the model has when training ends. With ``training.main.common.load_best_state_dict: true`` (default), training restores the best-validation weights first. When the run wrote exactly one best checkpoint, ``final_model.pt`` is then a hard link to it, not a second copy. A run with ``experiment.checkpointing.enabled: true`` and ``save_every_n_epochs: 1`` that trains for two epochs adds: .. code-block:: text main_network/ |-- standard_best_model.pt |-- standard_checkpoints/ | |-- epoch_0.pt | |-- epoch_1.pt | `-- epoch_2.pt `-- standard_loss_curves.png All ``.pt`` files are PyTorch state dictionaries. Load one into a model built from the same configuration: .. code-block:: python import torch state = torch.load("final_model.pt", map_location="cpu", weights_only=True) model.load_state_dict(state) ``dendritic-analyze-checkpoint`` rebuilds the model and the dataset splits from ``config.json`` and analyzes a checkpoint without training; see :ref:`checkpoint-analysis`. Resuming an interrupted run is described in :ref:`resuming-training`. FSDP runs ~~~~~~~~~ The FSDP entry point (``-m dendritic_modeling.scripts.training.train_experiments_fsdp``) writes the same configuration and environment records and ``final_model.pt``, a plain state dictionary of the gathered model. Its trainer saves checkpoints, which also hold the optimizer state and the epoch, under ``checkpoints/`` in the run directory, and only with ``experiment.checkpointing.enabled: true``: * ``checkpoints/best_model.pt`` after each improvement of the validation loss (with ``training.main.common.early_stopping: true``); * ``checkpoints/checkpoint_epoch_.pt`` every ``save_every_n_epochs`` epochs, where ```` counts epochs from zero; * ``checkpoints/final_checkpoint.pt`` after the last epoch. See :ref:`distributed-training`. .. _logging: Logging ------- Importing ``dendritic_modeling`` configures no logging: the package loggers carry only a ``NullHandler`` and write no files. Every installed command (``dendritic-train``, ``dendritic-sweep``, ``dendritic-analyze-checkpoint``, the ``dendritic-collect-*`` collectors and the others) calls ``dendritic_modeling.utils.logging_config.setup_logging()``, which prints the package's records to standard output, at INFO level or, with ``debug=True``, DEBUG level. ``setup_logging`` also writes the records to ``dendritic_modeling.log`` when it is given a ``log_dir`` or when the ``DENDRITIC_LOG_DIR`` environment variable is set; a relative path resolves against the current directory. When a program runs as a script, with ``python -m`` or under ``torchrun``, its own module logs as ``__main__``, outside the package's logger tree. ``setup_logging`` attaches the same handlers to that logger, so the program's records reach the console and the log file as well. Analyzers log under ``dendritic_modeling.analysis.``, where ```` is the analyzer's class name or the custom name it passes to ``AbstractAnalyzer`` (a name already inside ``dendritic_modeling`` is kept), so analyzers defined outside the package, in plugins for example, also write to the console and to ``dendritic_modeling.log``. A training run writes two log files into its run directory: ``dendritic_modeling.log`` The complete log of the run: the package's and the program's records at DEBUG level and above. ``train.log`` The run-setup records (run directory, data root, dataset sizes, model class). A later run in the same process closes the earlier run's log files. A program that calls the library from Python configures logging the same way, by calling ``setup_logging()`` or with the standard ``logging`` module. Exit status ----------- Every installed command exits with status 0 when it succeeds. An error raises a Python exception, which prints a traceback and exits with status 1; invalid command-line arguments exit with status 2. ``dendritic-analyze`` prints its error instead of a traceback (add ``--verbose`` for the traceback) and also exits with status 1. The Python ``main`` functions of the analysis commands return their results to a caller. A run that fails can leave a partial run directory: the records written before the failure (configuration, seeds, environment, logs) stay, and the log files contain the error. Requests that the configuration alone rules out fail before the run directory is created: options the loader rejects, and autoencoder encoders without pretraining or a trained checkpoint (:ref:`encoder-pretraining`). .. _run-reproducibility: Reproducibility --------------- Each run directory records what is needed to repeat the run: * ``config.json``: the complete resolved configuration; * ``resolved_seeds.json``: every random stream and the numerics settings; * ``runtime_environment.json``: the library commit and the versions of the numerical dependencies; * ``dataset_fingerprints.json`` (optional): digests of the dataset splits. In ``runtime_environment.json``, ``library.imported_version`` is the version declared by the imported ``dendritic_modeling`` module. ``library.distribution_version`` is the installed distribution's metadata version, or ``null`` when no distribution metadata is available. These can differ when ``PYTHONPATH`` selects a checkout other than the installed package; use the imported path and Git commit to identify the executed source. Source-identity records made with ``include_package=True`` also include ``package.imported_version``; their existing ``package.version`` field retains the distribution-metadata value for compatibility. Older receipts are unchanged and may lack the imported-version field. To repeat a run, check out the recorded commit, install the same dependency versions (``setup/freeze_constraints.py`` writes a constraints file for an environment; see :doc:`installation`), and train ``config.json`` on the same kind of device. Agreement to the last bit also needs the same device, batch size and determinism settings. :ref:`random-seeds` describes the seed streams, ``experiment.strict_deterministic`` and the numerics flags. :doc:`project_provenance` describes how a project pins its own code to an exact library commit, runs it from that commit and records what ran.