Runs and outputs

Every dendritic-train run writes its configuration, its records and its results into one run directory. This page describes where that directory is, every file a run writes, how the commands log, how they report failure, and which records let you repeat a run. Quickstart shows a complete run; Command-line reference lists every command-line option.

Run directory layout

Run directory name

A run writes into

<outputs.results_dir>/<outputs.run_name>_<YYYYMMDD_HHMMSS>/

where the timestamp is the local time at which the run starts. The defaults are outputs.results_dir: results and outputs.run_name: ei_sweep, so a config without an outputs block writes to results/ei_sweep_<timestamp>/. Set both in every config:

outputs:
  results_dir: outputs
  run_name: population_network_feedforward

dendritic-train --output_dir <dir> replaces outputs.results_dir and --run_name <name> replaces outputs.run_name. A relative results_dir is resolved against the current directory, and it is created when it does not exist.

Two cases use results_dir itself as the run directory, without a timestamped subdirectory:

  • outputs.exact_run_dir: true. Tools that claim an output directory in advance set it: the configurations that dendritic-sweep generates write to <sweep directory>/results/config_<N> this way (see Parameter sweeps), and dendritic-analyze-checkpoint uses it for its --output-dir.

  • A results_dir whose last component is config_<N>, as in configurations generated by earlier sweep versions.

A run never shares a timestamped directory: when another run with the same results_dir and run_name started in the same second, the later run writes to <run_name>_<timestamp>_1 (then _2, …).

Under torchrun, rank 0 creates the run directory and writes the run’s files; the other ranks receive the directory’s path from rank 0.

Files in a run directory

This is the directory of a one-epoch CPU run of configs/examples/population_network_feedforward.yaml (see Quickstart):

population_network_feedforward_<YYYYMMDD_HHMMSS>/
|-- analysis_status.json
|-- config.json
|-- dendritic_modeling.log
|-- final_model.pt
|-- init_gate_stats.json
|-- main_network/
|   `-- standard_best_model.pt
|-- model_resources.json
|-- nparams
|-- operator_card.json
|-- performance/
|   `-- final.json
|-- post_calibration_gate_stats.json
|-- reactivation_calibration.json
|-- resolved_seeds.json
|-- runtime_environment.json
|-- train.log
`-- training_summary.json

The files, in the order a run writes them:

File

Contents and when it is written

train.log, dendritic_modeling.log

The run’s log files; see Logging.

resolved_seeds.json

Every random seed the run uses (seed, dataset_seed, split_seed, model_seed, topology_seed, loader_seed, evaluation_seed, probe_seed), strict_deterministic, and the numerics settings applied to PyTorch (numerics: float32 matmul precision, TF32, cuDNN determinism and benchmarking, deterministic algorithms). See Random seeds and reproducibility flags.

runtime_environment.json

What ran: the imported package path, the checkout’s commit, branch and a digest of uncommitted source changes, the Python and PyTorch builds (CUDA, cuDNN, threads, default dtype), the versions of NumPy, SciPy, scikit-learn, pandas, Triton, torchvision and OmegaConf, the visible devices, the process group, the thread, CUDA and DENDRITIC_* environment variables, the platform and the host name.

dataset_fingerprints.json

Digests of the training, validation and test splits. Written only with experiment.record_dataset_fingerprints: true.

init_gate_stats.json

Statistics of every reactivation gate’s parameters after the model is built, with warnings about suspicious values, and the initialization policy each dendritic layer applied to its gate (init_policy, counted under init_policies). Written by every run; for a model without gates it lists no modules.

config.json

The resolved configuration: every field of the typed schema with its default filled in, after the command-line overrides. It also records the seeds filled in for the analyzers, the gradient-clipping mode the strategy applies and the distributed mode the run trained with. dendritic-train and dendritic-analyze-checkpoint accept it in place of the original YAML file.

reactivation_calibration.json, post_calibration_gate_stats.json

The calibration of data-driven gates (a gate init_policy such as occupancy_quantile; see Gate initialization and maintenance) and the gate statistics after it. Written only when a gate uses a data-driven policy; a run that starts from model.initial_checkpoint skips the calibration.

initial_checkpoint_provenance.json

Written when the run starts from model.initial_checkpoint (Starting from saved weights): the checkpoint’s path and SHA-256 and the keys that did not match the model.

main_network/

Checkpoints of the main training phase; see Checkpoints.

nparams

JSON (without a file extension) holding the total parameter count, {"nparams": <count>}.

model_resources.json

Parameter counts (total, trainable, core, synapse candidates, active synapses), per-synapse-module records, the persistent recurrent state per sample, the device and, on CUDA, the peak memory.

operator_card.json

The operator axes of the built core: weight domain and transform, integration rule, gate type and initialization policy, morphology (branch factors, leak conductance) and synapse mechanism, plus the result of the biological_neuron check (see Biological constraints).

training_summary.json

The best epoch and its validation loss, the training and validation loss of every epoch (the training loss also split into its base and regularization terms), the data-loader settings and the data-parallel sizes (world size, batch size per rank, global batch size, steps per epoch). Model states are left out.

final_model.pt

The state dictionary of the final model; see Checkpoints.

analysis_status.json

The completion status (completed, skipped, failed or empty) of every analyzer each analysis pass was asked to run; written by the passes that run at least one analyzer. See Completion status.

<analyzer directory>/

One directory per enabled analyzer, for example performance/ with final.json (accuracy, AUC and log-likelihood on each split) and, with analysis.performance.training: true, epochs/epoch<N>.json and evolution plots. See Where results are written.

dendritic-train --validate-only builds the datasets, the model, the calibrated gates, the analysis manager, the optimizer and the trainer, then stops. It writes the configuration and construction records above (config.json, the seed, environment and gate files, nparams, model_resources.json, operator_card.json, the logs and an empty main_network/) and no training results.

training.main.strategy: presentation_stream also writes one line of metrics per epoch to main_network/metrics.jsonl (the file name is training.main.strategies.presentation_stream.metrics_file).

Checkpoints

Checkpoint file names start with a prefix that names the training phase, for example standard_ for training.main.strategy: standard.

main_network/<prefix>best_model.pt

The weights of the epoch with the lowest validation loss. training.main.common.save_best_checkpoint sets when it is written: every_improvement (default) after each epoch that improves the validation loss, end once after training, never not at all.

main_network/<prefix>checkpoints/epoch_<N>.pt

Written only with experiment.checkpointing.enabled: true, every experiment.checkpointing.save_every_n_epochs epochs (default 10). epoch_0.pt holds the weights before the first epoch. These files hold model weights only, not the optimizer state.

main_network/<prefix>loss_curves.png

The training and validation loss curves, written when training.main.common.plot_losses is true (default) and the run trains for more than one epoch.

final_model.pt

The weights the model has when training ends. With training.main.common.load_best_state_dict: true (default), training restores the best-validation weights first. When the run wrote exactly one best checkpoint, final_model.pt is then a hard link to it, not a second copy.

A run with experiment.checkpointing.enabled: true and save_every_n_epochs: 1 that trains for two epochs adds:

main_network/
|-- standard_best_model.pt
|-- standard_checkpoints/
|   |-- epoch_0.pt
|   |-- epoch_1.pt
|   `-- epoch_2.pt
`-- standard_loss_curves.png

All .pt files are PyTorch state dictionaries. Load one into a model built from the same configuration:

import torch

state = torch.load("final_model.pt", map_location="cpu", weights_only=True)
model.load_state_dict(state)

dendritic-analyze-checkpoint rebuilds the model and the dataset splits from config.json and analyzes a checkpoint without training; see Analyzing saved checkpoints. Resuming an interrupted run is described in Resuming from an epoch boundary.

FSDP runs

The FSDP entry point (-m dendritic_modeling.scripts.training.train_experiments_fsdp) writes the same configuration and environment records and final_model.pt, a plain state dictionary of the gathered model. Its trainer saves checkpoints, which also hold the optimizer state and the epoch, under checkpoints/ in the run directory, and only with experiment.checkpointing.enabled: true:

  • checkpoints/best_model.pt after each improvement of the validation loss (with training.main.common.early_stopping: true);

  • checkpoints/checkpoint_epoch_<E>.pt every save_every_n_epochs epochs, where <E> counts epochs from zero;

  • checkpoints/final_checkpoint.pt after the last epoch.

See Distributed training.

Logging

Importing dendritic_modeling configures no logging: the package loggers carry only a NullHandler and write no files. Every installed command (dendritic-train, dendritic-sweep, dendritic-analyze-checkpoint, the dendritic-collect-* collectors and the others) calls dendritic_modeling.utils.logging_config.setup_logging(), which prints the package’s records to standard output, at INFO level or, with debug=True, DEBUG level. setup_logging also writes the records to dendritic_modeling.log when it is given a log_dir or when the DENDRITIC_LOG_DIR environment variable is set; a relative path resolves against the current directory.

When a program runs as a script, with python -m or under torchrun, its own module logs as __main__, outside the package’s logger tree. setup_logging attaches the same handlers to that logger, so the program’s records reach the console and the log file as well.

Analyzers log under dendritic_modeling.analysis.<name>, where <name> is the analyzer’s class name or the custom name it passes to AbstractAnalyzer (a name already inside dendritic_modeling is kept), so analyzers defined outside the package, in plugins for example, also write to the console and to dendritic_modeling.log.

A training run writes two log files into its run directory:

dendritic_modeling.log

The complete log of the run: the package’s and the program’s records at DEBUG level and above.

train.log

The run-setup records (run directory, data root, dataset sizes, model class).

A later run in the same process closes the earlier run’s log files. A program that calls the library from Python configures logging the same way, by calling setup_logging() or with the standard logging module.

Exit status

Every installed command exits with status 0 when it succeeds. An error raises a Python exception, which prints a traceback and exits with status 1; invalid command-line arguments exit with status 2. dendritic-analyze prints its error instead of a traceback (add --verbose for the traceback) and also exits with status 1. The Python main functions of the analysis commands return their results to a caller.

A run that fails can leave a partial run directory: the records written before the failure (configuration, seeds, environment, logs) stay, and the log files contain the error. Requests that the configuration alone rules out fail before the run directory is created: options the loader rejects, and autoencoder encoders without pretraining or a trained checkpoint (Pretraining autoencoder encoders).

Reproducibility

Each run directory records what is needed to repeat the run:

  • config.json: the complete resolved configuration;

  • resolved_seeds.json: every random stream and the numerics settings;

  • runtime_environment.json: the library commit and the versions of the numerical dependencies;

  • dataset_fingerprints.json (optional): digests of the dataset splits.

In runtime_environment.json, library.imported_version is the version declared by the imported dendritic_modeling module. library.distribution_version is the installed distribution’s metadata version, or null when no distribution metadata is available. These can differ when PYTHONPATH selects a checkout other than the installed package; use the imported path and Git commit to identify the executed source. Source-identity records made with include_package=True also include package.imported_version; their existing package.version field retains the distribution-metadata value for compatibility. Older receipts are unchanged and may lack the imported-version field.

To repeat a run, check out the recorded commit, install the same dependency versions (setup/freeze_constraints.py writes a constraints file for an environment; see Installation), and train config.json on the same kind of device. Agreement to the last bit also needs the same device, batch size and determinism settings. Random seeds and reproducibility flags describes the seed streams, experiment.strict_deterministic and the numerics flags. Project provenance describes how a project pins its own code to an exact library commit, runs it from that commit and records what ran.