Distributed training
DendriNet trains on several GPUs with one of two PyTorch runtimes:
DDP (
DistributedDataParallel): every process holds a complete copy of the model and trains on its own shard of the training set; the gradients are averaged after every backward pass.dendritic-trainruns it whentorchrunlaunches it with several processes.FSDP (
FullyShardedDataParallel): the parameters, gradients and optimizer state are sharded across the processes, so a model larger than one GPU can train. The separate entry pointdendritic_modeling.scripts.training.train_experiments_fsdpruns it, with a plain backpropagation loop.
Both runtimes start one process per GPU and use the NCCL process-group backend, so they need CUDA GPUs.
|
Entry point |
When to use it |
|---|---|---|
|
|
The model fits on one GPU; one process. |
|
|
The model fits on one GPU and more GPUs should process more samples per step. Runs most strategies. |
|
|
The model does not fit on one GPU. Runs |
Launching a run
torchrun starts the processes. On one node with four GPUs:
# DDP
torchrun --nproc_per_node=4 \
-m dendritic_modeling.scripts.training.train_experiments \
configs/my_config.yaml
# FSDP
torchrun --nproc_per_node=4 \
-m dendritic_modeling.scripts.training.train_experiments_fsdp \
configs/config_exp_fsdp.yaml
On several nodes, start torchrun on every node with the node count, the
node’s rank and the address of the first node:
torchrun --nnodes=2 --node_rank=0 --nproc_per_node=4 \
--master_addr=<first-node-address> --master_port=29500 \
-m dendritic_modeling.scripts.training.train_experiments_fsdp \
configs/my_config.yaml
Both entry points take the options of dendritic-train
(Command-line reference), such as --output_dir, --seed and
--epochs. --validate-only under torchrun builds the datasets and
the model, wraps the model (DDP or FSDP), builds the optimizer and the
trainer, writes the construction records and exits without training. The
launcher and the scheduler allocation set the number of nodes and GPUs, not
the configuration file.
For Slurm clusters the library includes job helpers:
# Write a Slurm script for one run; --dry-run prints it, --submit submits it
python -m dendritic_modeling.scripts.tools.generate_slurm_job \
configs/config_exp_fsdp.yaml --account ACCOUNT --partition PARTITION --dry-run
dendritic_modeling.scripts.tools.generate_slurm_jobwrites a script for the entry point thatdistributed.modeselects: one GPU withpythonfornone, and one process per GPU withtorchrunon one node forddp(--gpus-per-node, at least 2, is required) andfsdp(by default the smallest power of two GPUs whose sharded memory estimate fits--gpu-memory-gb, at most--max-gpus-per-node).--accountand--partitionare required.dendritic_modeling.scripts.tools.submit_fsdp_jobdoes the same for an FSDP configuration and submits by default (--generate-onlywrites the script only).dendritic_modeling.scripts.tools.estimate_gpu_requirementsestimates the GPU memory of a configuration (--batch-size,--precision).scripts/train_fsdp.shin the repository is ansbatchscript that startstrain_experiments_fsdpundertorchrunon every node of the allocation, for multi-node FSDP runs. It takes the configuration and the number of GPUs per node, runs the checkout it is submitted from, and names no partition or account: pass them tosbatch.
Sweeps launch DDP and FSDP conditions through slurm_config (Parameter sweeps).
The distributed section
The distributed section of an FSDP configuration, with the two training
options that every FSDP run of a model with sparse synapses needs
(Options the FSDP trainer rejects):
distributed:
mode: fsdp # none | ddp | fsdp
timeout_seconds: 3600 # null: 600 s under train_experiments, 1800 s under train_experiments_fsdp
fsdp:
sharding_strategy: FULL_SHARD
auto_wrap_policy: indexed_synapses
mixed_precision: true
mixed_precision_dtype: bf16
training:
main:
common:
synapse_decay_rate: 0.0 # FSDP runs no sparse-synapse maintenance
weight_boosting: false
distributed.mode declares the runtime, and each entry point checks it
against its launch. train_experiments trains in one process (none) or
with DDP under torchrun (ddp); train_experiments_fsdp trains with
FSDP and needs torchrun. A mode the entry point cannot run raises:
fsdp under train_experiments, ddp under the FSDP entry point, and
ddp without torchrun. Otherwise the mode the run actually used (for
example none for mode: ddp launched with one process) replaces the
declared one, with a warning, and config.json records it. An unknown mode
fails when the configuration loads, and so does a top-level fsdp block:
FSDP settings are read only under distributed.fsdp
(dendritic-upgrade-config removes a top-level block that repeats them or
has use_fsdp: false, and reports any other).
distributed.timeout_seconds bounds how long a collective operation may
wait before the process group aborts the run. null keeps PyTorch’s NCCL
default of 600 seconds under train_experiments and uses 1800 seconds
under train_experiments_fsdp. Work that only rank 0 does runs while the
other ranks wait in their next collective: training-time analyses after each
DDP epoch, and the data-driven gate calibration before FSDP wraps the model.
Raise the timeout when those phases can take longer. Network settings of
NCCL (interfaces, InfiniBand) belong to the launch environment, not the
configuration.
Batches, samplers and validation
training.main.common.batch_size is per process: with W processes one
optimizer step sees W * batch_size samples, an epoch has 1/W as many
steps, and the learning rate is not rescaled. Schedules counted in optimizer
steps or forward passes (indexed rewiring, dense-to-sparse schedules)
therefore reach a given step at a different epoch when W changes.
training_summary.json records the layout under data_parallel
(world_size, batch_size_per_rank, global_batch_size,
train_steps_per_epoch), and runtime_environment.json records the
process group.
The training set is split across the ranks by a DistributedSampler seeded
with the loader seed; it reshuffles every epoch. Validation pads every rank to
the same number of batches (PaddedDistributedEvalSampler) and leaves the
padding out of the metrics, so the reported loss and accuracy are the
sample-weighted means over the validation set, as in a single-process run, up
to floating-point rounding and the configured mixed precision.
Each rank trains on its own batch. Synapses whose training forward pass keeps statistics of the batch (the input moments of variance Top-K, the forward impact and source activity of release synapses) reduce their batch sums and counts over all ranks under DDP and FSDP, one all-reduce per such synapse and training forward pass. Every rank then keeps the statistics of the global batch and makes the same selections and rewiring decisions, as one process training on the combined batch would.
DDP
dendritic-train wraps the model in DistributedDataParallel when
torchrun starts it with several processes. Rank 0 creates the run
directory and writes the run records, analyses and checkpoints; the other
ranks train. Data-driven gate calibration runs before training, and rank 0’s
calibrated gates are broadcast to every rank.
DDP registers the parameters that require gradients when it wraps the model, and waits for a gradient of each of them in every backward pass. Options that change this set after wrapping, or train without such a backward pass, raise before the first step:
freeze_layers,freeze_branches,freeze_branch_kl,multi_stageandpresentation_streamrun in a single process only.voltage_stabilization,homeostatic_controlandei_equilibriumraise, because their warm-up loss gives the decoder no gradient.reactivation_update_mode: quantileorfrozenraises for the strategies whose gradients DDP reduces (standard,recurrent,soma_dfa).fa,dfaandlocal_cacompute their gradients themselves, average them across the ranks before clipping, and accept it.training.main.common.impact_rewiringraises.credit_deepstsynapses raise (their rewiring uses credit recorded before gradients are averaged), and so do DeepST synapses without a topology seed (model.core.sparsity.indexed.seed, orindexed_seedin a population network), which would rewire differently on each rank.
The supported strategies are standard, recurrent, fa, dfa,
soma_dfa and local_ca (Training strategies). Feedback-alignment
matrices are drawn from the run seed and the block names, so every rank
builds the same ones. MuonH relies on DDP averaging the gradients before its
nonlinear update; for the first muon_rank_consistency_check_steps steps it
compares the updated matrices across the ranks and raises if they differ.
FSDP
train_experiments_fsdp sets a run up as train_experiments does: the
same command-line options, configuration checks, run directory, seeds,
datasets, model (including model.initial_checkpoint, which every rank
loads before wrapping) and run records, so one configuration builds the same
model under both entry points. The model stays on the CPU until FSDP wraps
it, and each wrapped unit moves to its GPU as it is sharded. The FSDP trainer
builds its optimizer from training.main.optimizer and the parameter groups
of single-process training (trainable parameters only), so every parameter
trains at the rate it would have without FSDP (Parameter groups and learning rates).
FSDP options
distributed.fsdp takes these options:
Option |
Default |
Meaning |
|---|---|---|
|
|
|
|
|
How the model is split into FSDP units: |
|
|
Smallest unit of |
|
|
Compute in reduced precision. |
|
|
The dtype of mixed precision for parameters, gradient reduction and
buffers, unless set separately: |
|
|
|
|
|
Keep the sharded parameters in CPU memory (slower, less GPU memory). |
|
|
Broadcast rank 0’s parameters and buffers when wrapping. |
|
|
Keep the original parameters, which the parameter groups hold. |
|
|
Passed to |
|
|
|
|
|
Recompute activations in the backward pass (Memory). |
|
|
Readout-only training: freeze every core parameter before wrapping; the core still runs its full forward dynamics. |
|
|
With |
|
|
Batch size of the epochs that use the cached core outputs. |
Options the FSDP trainer rejects
The FSDP loop is plain backpropagation with an optimizer step, so the entry
point raises ValueError, listing every problem, for options that it would
not apply:
Option |
Reason |
|---|---|
|
The loop is backpropagation. |
|
There is no learning-rate scheduler; |
|
No auxiliary losses and no pruning step. |
|
No gate recalibration. |
|
No sparse-synapse maintenance. The default |
Rewiring synapses ( |
No rewiring step and no topology controllers. |
|
The trainer keeps its final weights and tracks no best state. |
|
The FSDP sampler always shuffles. |
|
Not implemented in the FSDP step. |
|
Sharded units expose 1-D views of the matrices that MuonH orthogonalizes. |
|
fp16 gradients underflow without the sharded gradient scaler. |
|
The parameter groups hold module parameters, which FSDP would replace with flat parameters. |
Parameters with zero dimensions |
FSDP shards only parameters with at least one dimension. The learned
global output scale ( |
Losses that need the model itself, the sequence losses and autoencoder
encoders also raise (Loss functions). Because the default
synapse_decay_rate is positive, an FSDP configuration of a model with
sparse synapses sets synapse_decay_rate: 0.0 and weight_boosting:
false explicitly, as the example in The distributed section
does; configs/config_exp_fsdp.yaml is a complete FSDP configuration of a
population network with indexed synapses. The FSDP trainer clips the global
gradient norm unless grad_clip_mode says otherwise. It keeps the final
weights: with load_best_state_dict: true it logs a warning and records
final_weights, load_best_state_dict_requested and
load_best_state_dict_applied in training_summary.json.
log_first_step_diagnostics: true records statistics of the first
optimizer step.
Wrapping and construction
auto_wrap_policy decides the FSDP units. With none the whole model is
one unit, so every GPU holds the complete model while it computes.
indexed_synapses makes every indexed synapse module (indexed and
indexed_dynamic synapses and their subclasses) its own unit, so the
parameter all-gathers of a contact-heavy model stay bounded; size wraps
every submodule with at least min_num_params parameters.
Data-driven gate calibration (for example occupancy_quantile) runs on rank
0 before wrapping, whether or not analyses are enabled, and reaches the other
ranks through sync_module_states; a data-driven policy with
sync_module_states: false raises.
Every rank must hold the same indexed topology. With the default
init_mode: per_rank each rank samples it from the same seed; with
init_mode: rank0_broadcast (model.core.sparsity.indexed.init_mode, or
indexed_init_mode in a population network) only rank 0 samples it, and
the other ranks receive it when FSDP wraps the model with
sync_module_states: true. Layers with persistent_indices: false
regenerate their indices from an explicit seed when a checkpoint loads and
check them against a stored topology_fingerprint
(Compact topology and checkpoints).
Mixed precision
Mixed precision defaults to bf16. fp16 needs loss scaling
(training.main.common.use_amp: true) and raises without it. FSDP casts
every floating buffer to buffer_dtype for the rest of the run and keeps no
full-precision copy, so buffers updated in place (variance Top-K statistics)
lose precision and checkpoints store the rounded values; buffer_dtype:
keep leaves buffers in their own dtype. Recurrent layers compute their
decay factors \(\exp(-\Delta t/\tau)\) in the parameter dtype whatever
buffer_dtype is: in bf16 every \(\tau\) above about 511 steps gives a
factor of exactly 1, and that level stops responding to input. The entry point
warns about such buffers (use mixed_precision_dtype: fp32 for these time
constants) and records them under frozen_decay_buffers, with the resolved
dtypes under fsdp_mixed_precision, in training_summary.json.
Checkpoints and final weights
At the end of training the ranks gather the weights once; rank 0 saves them
as a plain state dict, final_model.pt in the run directory, as
single-process training does, and runs the final analyses on an unsharded copy
of the model. The FSDP trainer runs no analyses during training, so analyzers
with training: true run only on the final model.
With experiment.checkpointing.enabled: true rank 0 also writes, under
<run_dir>/checkpoints/:
checkpoint_epoch_<e>.pteverysave_every_n_epochsepochs, whereecounts epochs from 0; none is written after the last epoch;best_model.ptat every validation improvement, whenearly_stoppingis true;final_checkpoint.ptat the end of training.
These files come from training.fsdp_utils.save_fsdp_checkpoint: the
unsharded model and optimizer state, the epoch, and the random generators of
every rank (Python, NumPy, CPU and the rank’s CUDA device).
training.fsdp_utils.load_fsdp_checkpoint(model, optimizer, path) restores
them into a newly wrapped model and optimizer and returns the epoch, so
training continued on the same number of ranks draws the same dropout masks
and stochastic synapse selections as the uninterrupted run. A checkpoint from
a different number of ranks, or one without generator state, raises unless
restore_random_state=False. The training scripts have no resume option:
call load_fsdp_checkpoint from your own launcher.
Memory
When a run does not fit, in roughly this order:
Reduce
training.main.common.batch_size.Use
auto_wrap_policy: indexed_synapses(orsize) so that no GPU holds the whole model at once.Keep
sharding_strategy: FULL_SHARD, which shards parameters, gradients and optimizer state.Enable
distributed.fsdp.gradient_checkpointing: true. It recomputes, in the backward pass, the outermost dendritic branch layers, E/I layers and networks, andnn.Linearlayers with more than 1000 inputs. Models whose synapses change state in their training forward pass (dense-to-sparse schedules advanced on the forward pass, variance Top-K, release synapses) reject it, because the recomputation would replay those updates and compute gradients for a different mask.Enable
distributed.fsdp.cpu_offload: true(slower).Train only the readout with
freeze_core: trueand, for recurrent cores,cache_frozen_core_outputs: true.
The entry point records the peak GPU memory, the maximum over the ranks, in
model_resources.json.
Troubleshooting
- NCCL timeouts
Raise
distributed.timeout_secondswhen rank 0 runs long analyses or calibration while the other ranks wait. If NCCL selects the wrong network interface, setNCCL_SOCKET_IFNAME(and, where needed,NCCL_IB_DISABLE) in the launch environment, and check the network connectivity between the nodes.- Out of GPU memory
See Memory.
- An option is rejected at start
The FSDP entry point lists every option it would not apply in one error; remove them or train with
train_experiments. DDP rejects the strategies and options listed under DDP.- Autoencoder encoders
Encoder pretraining (Pretraining autoencoder encoders) runs in single-process training. A DDP run starts from the checkpoint of a single-process run (
model.initial_checkpoint); the FSDP entry point rejects autoencoder encoders.- Monitoring
Rank 0 writes the run’s log files,
dendritic_modeling.logandtrain.log, into the run directory (Logging). Watch GPU memory withnvidia-smi; when a scheduler runs the job, follow its job output file.