Analysis
Analyzers measure a trained model: its performance, the information its
compartments carry, the effect of interventions, its connectivity and its
dynamics. Each analyzer has a section analysis.<name> in the experiment
configuration. Enabled analyzers run on the final model of a training run,
and optionally after every epoch, and write their results into the run
directory. Analyzer reference describes every analyzer, and
Analyzing saved checkpoints shows how to run analyzers on saved checkpoints and
how to collect and summarize the results of many checkpoints.
Configuring analyses
Every section has three keys:
enabled(defaultfalse)Run the analyzer. All analyzers are disabled by default.
training(defaultfalse, except where the catalogue says otherwise)Also run it after every training epoch (see When analyzers run).
paramsThe analyzer’s options, documented in Analyzer reference.
routinghas noparams.
analysis.performance also has training_splits and final_splits,
and analysis.runtime sets how analyzers iterate over data
(Data iteration: analysis.runtime).
analysis:
performance:
enabled: true
training: true
training_splits: ["validation"]
final_splits: ["train", "validation", "test"]
weight:
enabled: true
information:
enabled: true
params:
selection:
components: ["basic", "conditional"]
Section names are the analyzer names in the Analyzer catalogue, which
are also the field names in a run’s saved config.json. Unknown section
names and unknown keys raise when the configuration loads.
The older section names ending in _analysis (for example
performance_analysis) and spike_trains still load, with a warning; a
section may not be written under both names.
When analyzers run
- After training
Every enabled analyzer runs once on the model the run ends with (the restored best-validation weights when
training.main.common.load_best_state_dictis true), whatever itstrainingflag. Its results are namedfinal.- During training
training: truealso runs the analyzer after every training epoch, with results namedepoch<N>(Ncounts from 1). There is no interval option, so per-epoch analyses add their cost to every epoch.
Two analyzers default to training: true: synapse_turnover, which
compares snapshots taken during training and raises with training: false,
and reactivation_dynamics. Ten analyzers are final-only and raise with
training: true: representation_accessibility,
source_tuning_support, input_region_intervention,
synapse_pruning, path_matched_intervention,
branch_local_sensitivity, node_perturbation_credit,
local_inhibitory_population_intervention, path_matched_information
and gain_load_perturbation. Both errors are
raised before training starts.
Other rules:
Multi-stage training. With
training.main.strategy: multi_stagethe analyzers run after each stage instead of after training (whentraining.main.strategies.multi_stage.analyze_between_stagesis true, the default). Each stage writes into<run>/<stage name>/and names its results after the stage; per-epochperformanceresults of all stages are also gathered, renumbered, into<run>/performance/epochs/.Distributed training. Under DDP the analyzers run in the main process only. Under FSDP per-epoch analyses are skipped, and the final analysis runs in the main process on an unsharded copy of the trained model.
Sealed test split.
experiment.sealed_test: truerejects every enabled analyzer (see Training, validation and test splits).Regression models. For a regression model
performancereplacesaccuracy,aucandcategorical_loglikelihoodbymseandcosine_similarity.Device. Analyses run on the GPU when one is visible and on the CPU otherwise. An analyzer may move the model, and restores its device and train/eval mode afterwards.
Random state. Every analysis pass runs in an isolated random state seeded with
experiment.evaluation_seed(informationalso withexperiment.probe_seed), so enabling analyzers does not change the training random stream (see Seeds).Models an analyzer does not support. Most analyzers that need a particular core (an E/I dendritic core, a recurrent core, spiking somas) skip other models, often with only a log message and without writing files; some raise. Analyzer reference lists the requirement of each. Every pass records whether each analyzer completed (Completion status).
Completion status
Every pass records the status of each analyzer it was asked to run (the
enabled analyzers; during training, those with training: true) in
analysis_status.json in the run directory (in a multi-stage run, in the
stage directory); a pass with no analyzer to run writes no record. The
status of an analyzer is one of:
completedThe analyzer returned and wrote at least one file into its directory, or reported that it keeps its results for a later pass (
synapse_turnoverduring training, recorded underdeferred_output).skippedThe analyzer did not run, with the
reason: it reported that it does not apply to the model or data, the pass stopped before reaching it, or the manager never runs it (a plugin analyzer registered without a phase).failedThe analyzer raised;
error_typeanderror_messagerecord the error, which still ends the pass and the run.emptyThe analyzer returned without writing any file into its directory.
A completed analyzer can list missing_outputs: optional outputs it did not
produce, each with the reason (for example a figure that needs inhibitory
statistics on a tree without inhibition). The file holds one record per pass,
under passes.<name> (final, a stage name, or the latest epoch<N>:
per-epoch passes keep only the most recent record), with requested (the
analyzers asked for), finished (false while the pass runs, or when an
error stopped it) and complete (every requested analyzer completed):
{
"schema_version": "analysis_status_v1",
"passes": {
"final": {
"training": false,
"require_completion": false,
"requested": ["performance", "correlation"],
"finished": true,
"complete": false,
"analyses": {
"performance": {"status": "completed", "output_dir": "performance"},
"correlation": {
"status": "skipped",
"output_dir": "correlation_analysis",
"reason": "the model has no ExcitationInhibitionNetwork core"
}
}
}
}
}
AnalysisManager.run_analysis returns the record of its pass. With
analysis.require_completion: true (default false) a pass whose
analyses did not all complete raises AnalysisIncompleteError after
writing the record, so a training run or a checkpoint analysis fails instead
of leaving an incomplete result directory. Result files are the same with
and without the record.
analysis:
require_completion: true
performance:
enabled: true
correlation:
enabled: true
Data iteration: analysis.runtime
analysis.runtime.training applies to the per-epoch runs and
analysis.runtime.final to the run after training. Both have the same keys:
Key |
Default |
Meaning |
|---|---|---|
|
see below |
|
|
|
Batch size when streaming. |
|
|
DataLoader worker processes when streaming. |
|
|
Pinned host memory; ignored when the analysis runs on the CPU. |
|
|
Keep DataLoader workers between passes. |
|
|
Batches prefetched per worker. |
|
|
|
|
|
Cap on the number of batches per split. |
|
|
Cap on the number of examples an analyzer sees (the first examples of the split). |
|
|
Size threshold of |
|
|
Seed of the analysis data loaders and evaluations; unset takes
|
mode defaults to materialize in single-process runs
(distributed.mode: none) and to auto under DDP and FSDP. Keys left out
of a written phase keep these defaults, including the mode of the
distributed mode. In materialize mode, and when auto materializes,
hook-based analyzers process the split in one batch.
On a CUDA device, a split whose one-batch pass certainly cannot fit is
streamed instead (Activation memory of one-batch passes).
analysis:
runtime:
training:
mode: stream
batch_size: 512
max_samples: 2048
final:
mode: auto
batch_size: 256
Every analyzer that reads data uses the phase’s profile, with these
exceptions: routing iterates the test split with batches of 128,
reactivation_dynamics takes one batch of params.max_samples examples,
and synapse_turnover, synapse_pruning, inhibitory_specialization,
jacobian_spectrum and dendritic_timetraces read no data. Analyzers
with their own sample option (for example params.max_samples) use the
smaller of the two caps. performance does not apply max_samples, and
applies max_batches only when it streams.
A training_runtime or final_runtime block under
analysis.performance (an older layout) is moved to analysis.runtime
when that section is absent and ignored, with a warning, otherwise; its
splits list becomes training_splits or final_splits.
Activation memory of one-batch passes
The size of a split does not bound the memory of a forward pass over it:
activations grow with the number of examples. Before a CUDA device runs a
materialized split in one forward pass, the library bounds the memory of that
pass from below: a copy of the model runs on the first 2, 4 and 8 examples,
with the random state restored, and the bytes of the tensors it holds at once
are extrapolated to the whole split. When the bound plus the split’s own
tensors exceed what the device can still provide (its free memory, PyTorch’s
unused cache, and at most a per-process memory fraction), the split is
streamed in batch_size batches instead of first failing with an
out-of-memory error (an INFO record says so). A pass that fits is unchanged:
it keeps its one batch and gives the same numbers.
The bound covers the scoring of the model through the shared metric
evaluation, used by performance, noise_perturbation,
single_layer_contribution, multiplicative_gain,
gain_load_perturbation and ablation (its scores without signal
interventions; the shuffle and mean_clamp interventions use forward
hooks, which disable the bound), and the passes of branch_activation,
synaptic_activation and branch_local_sensitivity. Analyzers that
materialize a split and run their own pass over it are not covered and can
still run out of memory on a split too large for one batch: information,
correlation, compartment_statistics, compartment_snr,
receptive_field, weight (source-region alignment),
source_tuning_support, representation_accessibility,
path_matched_information, path_matched_intervention,
local_inhibitory_population_intervention, input_region_intervention,
local_rule_components, spike_train and the reference pass of
ablation’s mean_clamp. Set mode: stream (or a smaller
materialize_threshold_mb with mode: auto, or max_samples) in the
runtime profile of such runs.
No bound is used, and the pass runs as before (one full batch; a metric whose pass fails is evaluated again by streaming, while an analyzer’s own pass raises) in these cases:
on the CPU, where the memory available to a process (Slurm limits, page cache) cannot be measured reliably enough to prove that a pass fails;
when forward hooks are registered, since the probes would run them;
for models with
workspace_mbchunking, and for modules that setbatch_adaptive_memory = True(a module defined outside the library sets it when its allocations do not grow linearly with the batch size, for example when it picks its chunk size from the batch size);when the probes do not grow linearly with the batch size;
when a probe fails for any reason, including an out-of-memory error while copying the model.
The probes cost one copy of the model on the device (the memory of its
parameters and buffers, freed afterwards) and four forward passes on at most
8 examples under a recording dispatch mode: about 0.07 s for a 14-million
parameter E/I network on an A100 once CUDA is warmed up, and about 0.4 s on
the first call of a process. They are reused for the same model, input shape
and device until a parameter or buffer of the model changes, so they run once
per evaluation pass of a fixed model, again after every training epoch (the
weights change), and again for every lesion of ablation.
Data splits
Each analyzer reads a fixed split, listed in the Analyzer catalogue.
Most read the test split. performance reads the splits in
training_splits (during training) and final_splits (after training),
both defaulting to all three. representation_accessibility fits its
probes, and path_matched_information estimates information, on the
validation split. Fixed reference statistics (for example the mean_clamp
reference of ablation) come from the training split.
Seeds
Seeds written in an analysis section are used unchanged. Every unset seed is filled when the analyzers are built:
analysis.runtime.<phase>.seedtakesexperiment.evaluation_seed;every other unset field named
seedor ending in_seedtakes a seed derived fromexperiment.probe_seedand the field’s configuration path, so each analyzer has its own stream, independent of which other analyzers are enabled.
Both experiment seeds fall back to experiment.seed (see
Random seeds and reproducibility flags). The filled values are saved in the run’s
config.json. Because the derivation uses the configuration path, two
analyzers never share a derived seed, and runs whose experiment.probe_seed
differs (by default, runs with different experiment.seed) draw different
analysis seeds. Write a seed explicitly when an analysis must use the same
random numbers in every checkpoint (for example
gain_load_perturbation.params.base_draw_seed), and write the same
support_seed (and support_namespace and support_draws) in both
sections to make path_matched_intervention and
path_matched_information select the same supports.
Where results are written
An analyzer writes into <run directory>/<directory>/, where the run
directory is <outputs.results_dir>/<outputs.run_name>_<YYYYmmdd_HHMMSS>/
(see Run directory layout) and <directory> is the analyzer’s directory in the
Analyzer catalogue. A result is named after its pass: final,
epoch<N> or a multi-stage stage name; this page writes <name> for it.
The directory is created when the analyzer first writes, so an analyzer that
skips the model leaves no directory.
The JSON result files follow four patterns:
<name>.json:performance,representation_accessibility,source_tuning_support,input_region_intervention,ablation,path_matched_intervention,branch_local_sensitivity,node_perturbation_credit,local_inhibitory_population_intervention,path_matched_information,compartment_snr,multiplicative_gain,gain_load_perturbation,inhibitory_specialization,jacobian_spectrum,dendritic_timetraces,spike_train.<name>(JSON without a file extension):information,weight,correlation,noise_perturbation,single_layer_contribution,compartment_statistics,routing,synaptic_activation,branch_activation,synapse_turnover,synapse_pruning.per-epoch results in an
epochs/subdirectory:performance,ablation,compartment_snrandmultiplicative_gainwriteepochs/epoch<N>.json;reactivation_dynamicswrites every pass,finalincluded, toepochs/<name>.json;local_rule_componentswritesepochs/epoch<N>.jsonand.csv, appends totrajectory.csv, and writesfinal/final.jsonand.csvafter training.other names:
adversarial_robustnesswritesadversarial_robustness_<name>.json, andreceptive_fieldwrites<name>_summary.json(inepochs/during training) and the raw maps<name>_raw_receptive_fields.pt.
Many analyzers also save figures (PNG or PDF; some have a save_figures or
figure-format option). Some figures use fixed file names and are replaced at
every pass, and some are drawn only after training, including animations of
per-epoch results (MP4, or GIF when ffmpeg is missing). Series are never
told apart by colour alone: each line has its own marker or line style, each
bar series its own hatch, and overlaid histograms their own outline style
(dendritic_modeling.plotting.styles).
Part of the directory of a two-epoch run in which performance,
information, ablation, reactivation_dynamics and
synapse_turnover ran with training: true (synapse_turnover only
takes snapshots during training and writes after it):
<run directory>/
ablation_analysis/
epochs/epoch1.json
epochs/epoch2.json
final.json
final_lesion_accuracy.png
information_analysis/
epoch1
epoch1_summary.txt
epoch2
epoch2_summary.txt
final
final_summary.txt
aggregated_basic_mi_comparison_depth.png
performance/
epochs/epoch1.json
epochs/epoch2.json
final.json
performance_evolution_accuracy.png
reactivation_dynamics/
epochs/epoch1.json
epochs/epoch2.json
epochs/final.json
synapse_turnover_analysis/
final
final_turnover.png
final_turnover_by_layer.png
Analyzer catalogue
Every analyzer registered in dendritic_modeling.analysis.registry. The
configuration block of each is analysis.<analyzer>. “During training”
says whether training: true is possible: “yes”, “no” (final-only), or the
default when it is true. “Directory” is the output directory inside the
run directory.
Analyzer |
Measures |
During training |
Data |
Directory |
|---|---|---|---|---|
Accuracy, AUC, log-likelihood, MSE, cosine similarity |
yes |
|
|
|
Scores under uniform and Gaussian input noise |
yes |
test |
|
|
Scores over a sweep of input gains |
yes |
test |
|
|
Scores over a factorial of shared gain and input load |
no |
test |
|
|
Accuracy under FGSM attacks |
yes |
test |
|
|
Scores with only one dendritic layer intact |
yes |
test |
|
|
Mutual information between compartment signals and classes |
yes |
test (or validation) |
|
|
Signal, noise and total correlations of E, I and output signals |
yes |
test |
|
|
Class discriminability of inputs, layers and branches |
yes |
test |
|
|
Probe decodability and information of soma populations |
no |
validation, test |
|
|
Class information on matched-width dendritic supports by depth |
no |
validation |
|
|
Score drops after lesioning, shuffling or clamping synapses |
yes |
test (reference: train) |
|
|
Score changes after replacing matched-width supports by depth |
no |
test (reference: train) |
|
|
Score changes after replacing one layer’s inhibitory population |
no |
test |
|
|
Effects of removing or keeping image regions |
no |
test (reference: train) |
|
|
Local derivatives of branch outputs by depth |
no |
test |
|
|
Compartment errors estimated from loss changes, against exact errors |
no |
test |
|
|
Weight statistics; optional exact source-support profiles |
yes |
none (alignment: validation) |
|
|
Class tuning of the inputs each sparse pathway selects |
no |
train, validation |
|
|
Contacts kept, formed and removed during training |
required (default |
none |
|
|
Census of realized contacts and of pruning steps |
no |
none |
|
|
Receptive-field maps of branch layers |
yes |
none (tuning options: test) |
|
|
Targeting of recurrent inhibition across dendritic levels |
yes |
none |
|
|
Statistics of weights, inputs and activations |
yes |
test |
|
|
Distributions of synaptic activations |
yes |
test |
|
|
Distributions of branch currents and outputs |
yes |
test |
|
|
Gate parameters and pre-gate voltage statistics |
default |
train during training, test after |
|
|
Activity shares of dendritic levels in recurrent cores |
yes |
test |
|
|
Singular values of recurrent weights per level |
yes |
none |
|
|
Simulated responses of the learned dendritic cascade |
yes |
synthetic input |
|
|
Firing statistics of spiking somas |
yes |
test |
|
|
Factors of the local learning rule by depth |
yes |
train during training, test after |
|
configs/templates/analysis_full_template.yaml writes out the options of
every analyzer except representation_accessibility,
source_tuning_support and input_region_intervention, which have their
own example files in configs/examples/ (see Analyzer reference).
Python interface
dendritic_modeling.analysis.AnalysisManager runs the configured analyzers
on a model outside the training scripts:
from dendritic_modeling.analysis import AnalysisManager
from dendritic_modeling.config import load_analysis_config
analysis_config = load_analysis_config("analysis_profile.yaml")
manager = AnalysisManager(
model,
data={"train": train_ds, "valid": valid_ds, "test": test_ds},
analysis_config=analysis_config,
save_root="analysis_output",
device="cpu",
evaluation_seed=42,
probe_seed=42,
)
manager.run_analysis(filename="final", training=False)
load_analysis_config(path, distributed_mode="none") reads a file whose
root is analysis: or that is the analysis mapping itself. save_root is
required. run_analysis writes files and returns None; read the results
from save_root. Without evaluation_seed and probe_seed unset seeds
stay unset and each analyzer uses its own default.
The package also exports analyzers that are used directly from Python rather
than configured in analysis: ClassAccessibilityProbeAnalyzer and
LinearClassAccessibilityAnalyzer (array-level probe decodability),
MatchedClassInformationAnalyzer (matched-support class information),
SameCheckpointCausalInterventionAnalyzer (phase, pathway or depth
interventions on one checkpoint), StateTransitionJacobianAnalyzer
(leading singular value of a recurrent core’s one-step state Jacobian),
InhibitoryRoutingComparison (seed-level records and paired summaries at
one contact budget) and PoissonExposureTransferAnalyzer, which
dendritic-poisson-exposure-transfer runs (see Analyzing saved checkpoints).
Adding an analyzer
An analyzer outside the library is a plugin component: its package registers
the analyzer with register_analyzer(..., phase=...) and its configuration
section analysis.<name> with register_config_section, and the manager
runs it in that phase of the order (Extending DendriNet).
An analyzer of the library itself needs:
a parameter and section dataclass, and a field of
AnalysisConfig, indendritic_modeling/config/analysis.py;an
AnalyzerSpecin_BUILTIN_ANALYZERS(dendritic_modeling/analysis/registry.py) with its output directory, constructor style and whether it supports or requires per-epoch runs;a step in the execution plan (
dendritic_modeling/analysis/managers/analysis_plan.py) that calls it with the right data;tests, an entry in
configs/templates/analysis_full_template.yamland a section in Analyzer reference.
See Extension registries for the other extension points.