Runs and outputs
Every dendritic-train run writes its configuration, its records and its
results into one run directory. This page describes where that directory is,
every file a run writes, how the commands log, how they report failure, and
which records let you repeat a run. Quickstart shows a complete run;
Command-line reference lists every command-line option.
Run directory layout
Run directory name
A run writes into
<outputs.results_dir>/<outputs.run_name>_<YYYYMMDD_HHMMSS>/
where the timestamp is the local time at which the run starts. The defaults
are outputs.results_dir: results and outputs.run_name: ei_sweep, so a
config without an outputs block writes to results/ei_sweep_<timestamp>/.
Set both in every config:
outputs:
results_dir: outputs
run_name: population_network_feedforward
dendritic-train --output_dir <dir> replaces outputs.results_dir and
--run_name <name> replaces outputs.run_name. A relative
results_dir is resolved against the current directory, and it is created
when it does not exist.
Two cases use results_dir itself as the run directory, without a
timestamped subdirectory:
outputs.exact_run_dir: true. Tools that claim an output directory in advance set it: the configurations thatdendritic-sweepgenerates write to<sweep directory>/results/config_<N>this way (see Parameter sweeps), anddendritic-analyze-checkpointuses it for its--output-dir.A
results_dirwhose last component isconfig_<N>, as in configurations generated by earlier sweep versions.
A run never shares a timestamped directory: when another run with the same
results_dir and run_name started in the same second, the later run
writes to <run_name>_<timestamp>_1 (then _2, …).
Under torchrun, rank 0 creates the run directory and writes the run’s
files; the other ranks receive the directory’s path from rank 0.
Files in a run directory
This is the directory of a one-epoch CPU run of
configs/examples/population_network_feedforward.yaml (see
Quickstart):
population_network_feedforward_<YYYYMMDD_HHMMSS>/
|-- analysis_status.json
|-- config.json
|-- dendritic_modeling.log
|-- final_model.pt
|-- init_gate_stats.json
|-- main_network/
| `-- standard_best_model.pt
|-- model_resources.json
|-- nparams
|-- operator_card.json
|-- performance/
| `-- final.json
|-- post_calibration_gate_stats.json
|-- reactivation_calibration.json
|-- resolved_seeds.json
|-- runtime_environment.json
|-- train.log
`-- training_summary.json
The files, in the order a run writes them:
File |
Contents and when it is written |
|---|---|
|
The run’s log files; see Logging. |
|
Every random seed the run uses ( |
|
What ran: the imported package path, the checkout’s commit, branch and
a digest of uncommitted source changes, the Python and PyTorch builds
(CUDA, cuDNN, threads, default dtype), the versions of NumPy, SciPy,
scikit-learn, pandas, Triton, torchvision and OmegaConf, the visible
devices, the process group, the thread, CUDA and |
|
Digests of the training, validation and test splits. Written only
with |
|
Statistics of every reactivation gate’s parameters after the model is
built, with warnings about suspicious values, and the initialization
policy each dendritic layer applied to its gate ( |
|
The resolved configuration: every field of the typed schema with its
default filled in, after the command-line overrides. It also records
the seeds filled in for the analyzers, the gradient-clipping mode the
strategy applies and the distributed mode the run trained with.
|
|
The calibration of data-driven gates (a gate |
|
Written when the run starts from |
|
Checkpoints of the main training phase; see Checkpoints. |
|
JSON (without a file extension) holding the total parameter count,
|
|
Parameter counts (total, trainable, core, synapse candidates, active synapses), per-synapse-module records, the persistent recurrent state per sample, the device and, on CUDA, the peak memory. |
|
The operator axes of the built core: weight domain and transform,
integration rule, gate type and initialization policy, morphology
(branch factors, leak conductance) and synapse mechanism, plus the
result of the |
|
The best epoch and its validation loss, the training and validation loss of every epoch (the training loss also split into its base and regularization terms), the data-loader settings and the data-parallel sizes (world size, batch size per rank, global batch size, steps per epoch). Model states are left out. |
|
The state dictionary of the final model; see Checkpoints. |
|
The completion status ( |
|
One directory per enabled analyzer, for example |
dendritic-train --validate-only builds the datasets, the model, the
calibrated gates, the analysis manager, the optimizer and the trainer, then
stops. It writes the configuration and construction records above
(config.json, the seed, environment and gate files, nparams,
model_resources.json, operator_card.json, the logs and an empty
main_network/) and no training results.
training.main.strategy: presentation_stream also writes one line of
metrics per epoch to main_network/metrics.jsonl (the file name is
training.main.strategies.presentation_stream.metrics_file).
Checkpoints
Checkpoint file names start with a prefix that names the training phase, for
example standard_ for training.main.strategy: standard.
main_network/<prefix>best_model.ptThe weights of the epoch with the lowest validation loss.
training.main.common.save_best_checkpointsets when it is written:every_improvement(default) after each epoch that improves the validation loss,endonce after training,nevernot at all.main_network/<prefix>checkpoints/epoch_<N>.ptWritten only with
experiment.checkpointing.enabled: true, everyexperiment.checkpointing.save_every_n_epochsepochs (default 10).epoch_0.ptholds the weights before the first epoch. These files hold model weights only, not the optimizer state.main_network/<prefix>loss_curves.pngThe training and validation loss curves, written when
training.main.common.plot_lossesis true (default) and the run trains for more than one epoch.final_model.ptThe weights the model has when training ends. With
training.main.common.load_best_state_dict: true(default), training restores the best-validation weights first. When the run wrote exactly one best checkpoint,final_model.ptis then a hard link to it, not a second copy.
A run with experiment.checkpointing.enabled: true and
save_every_n_epochs: 1 that trains for two epochs adds:
main_network/
|-- standard_best_model.pt
|-- standard_checkpoints/
| |-- epoch_0.pt
| |-- epoch_1.pt
| `-- epoch_2.pt
`-- standard_loss_curves.png
All .pt files are PyTorch state dictionaries. Load one into a model built
from the same configuration:
import torch
state = torch.load("final_model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state)
dendritic-analyze-checkpoint rebuilds the model and the dataset splits
from config.json and analyzes a checkpoint without training; see
Analyzing saved checkpoints. Resuming an interrupted run is described in
Resuming from an epoch boundary.
FSDP runs
The FSDP entry point (-m dendritic_modeling.scripts.training.train_experiments_fsdp)
writes the same configuration and environment records and final_model.pt,
a plain state dictionary of the gathered model. Its trainer saves
checkpoints, which also hold the optimizer state and the epoch, under
checkpoints/ in the run directory, and only with
experiment.checkpointing.enabled: true:
checkpoints/best_model.ptafter each improvement of the validation loss (withtraining.main.common.early_stopping: true);checkpoints/checkpoint_epoch_<E>.pteverysave_every_n_epochsepochs, where<E>counts epochs from zero;checkpoints/final_checkpoint.ptafter the last epoch.
See Distributed training.
Logging
Importing dendritic_modeling configures no logging: the package loggers
carry only a NullHandler and write no files. Every installed command
(dendritic-train, dendritic-sweep, dendritic-analyze-checkpoint,
the dendritic-collect-* collectors and the others) calls
dendritic_modeling.utils.logging_config.setup_logging(), which prints the
package’s records to standard output, at INFO level or, with debug=True,
DEBUG level. setup_logging also writes the records to
dendritic_modeling.log when it is given a log_dir or when the
DENDRITIC_LOG_DIR environment variable is set; a relative path resolves
against the current directory.
When a program runs as a script, with python -m or under torchrun,
its own module logs as __main__, outside the package’s logger tree.
setup_logging attaches the same handlers to that logger, so the program’s
records reach the console and the log file as well.
Analyzers log under dendritic_modeling.analysis.<name>, where <name>
is the analyzer’s class name or the custom name it passes to
AbstractAnalyzer (a name already inside dendritic_modeling is kept),
so analyzers defined outside the package, in plugins for example, also write
to the console and to dendritic_modeling.log.
A training run writes two log files into its run directory:
dendritic_modeling.logThe complete log of the run: the package’s and the program’s records at DEBUG level and above.
train.logThe run-setup records (run directory, data root, dataset sizes, model class).
A later run in the same process closes the earlier run’s log files. A program
that calls the library from Python configures logging the same way, by
calling setup_logging() or with the standard logging module.
Exit status
Every installed command exits with status 0 when it succeeds. An error raises
a Python exception, which prints a traceback and exits with status 1; invalid
command-line arguments exit with status 2. dendritic-analyze prints its
error instead of a traceback (add --verbose for the traceback) and also
exits with status 1. The Python main functions of the analysis commands
return their results to a caller.
A run that fails can leave a partial run directory: the records written before the failure (configuration, seeds, environment, logs) stay, and the log files contain the error. Requests that the configuration alone rules out fail before the run directory is created: options the loader rejects, and autoencoder encoders without pretraining or a trained checkpoint (Pretraining autoencoder encoders).
Reproducibility
Each run directory records what is needed to repeat the run:
config.json: the complete resolved configuration;resolved_seeds.json: every random stream and the numerics settings;runtime_environment.json: the library commit and the versions of the numerical dependencies;dataset_fingerprints.json(optional): digests of the dataset splits.
In runtime_environment.json, library.imported_version is the version
declared by the imported dendritic_modeling module.
library.distribution_version is the installed distribution’s metadata
version, or null when no distribution metadata is available. These can
differ when PYTHONPATH selects a checkout other than the installed package;
use the imported path and Git commit to identify the executed source.
Source-identity records made with include_package=True also include
package.imported_version; their existing package.version field retains
the distribution-metadata value for compatibility. Older receipts are unchanged
and may lack the imported-version field.
To repeat a run, check out the recorded commit, install the same dependency
versions (setup/freeze_constraints.py writes a constraints file for an
environment; see Installation), and train config.json on the same
kind of device. Agreement to the last bit also needs the same device, batch
size and determinism settings. Random seeds and reproducibility flags describes the seed
streams, experiment.strict_deterministic and the numerics flags.
Project provenance describes how a project pins its own code to an
exact library commit, runs it from that commit and records what ran.