Baselines and comparison models
DendriNet includes conventional networks to compare with dendritic cores:
point-neuron controls matched in size to a dendritic reference, dense and
sparse affine MLPs, and recurrent baselines. They are core types, selected
with model.core.type like the dendritic cores (Core types lists
every type), and they train with the same training strategies. The second
half of the page describes Python utilities for comparisons at a fixed
parameter budget and a few numerical helpers.
Controls matched to a dendritic reference
These core types build a point-neuron network whose size follows a dendritic
reference core. The reference is described by the same sections a dendritic
core uses: the structured E/I sections model.core.architecture and
model.core.connectivity (with morphology, sparsity and the other
structured sections; see Structured E/I cores), or, where the table
says so, a model.core.population_network block. The builder constructs the
reference only to count it; the control contains no dendritic operations.
|
Network |
Reference |
|---|---|---|
|
One hidden layer of point neurons, |
Structured E/I sections or |
|
With a structured reference: a dense MLP with as many linear layers as
the reference has dendritic branch layers in total (E and I trees of
every layer), equal hidden widths, a sigmoid after every layer and at
least as many weights as the reference has trainable parameters. With
a |
Structured E/I sections or |
|
As |
Structured E/I sections or |
|
A sparse structured MLP: the last layer of the reference rebuilt as
signed Top-K synapses on the same branch structure, with signed branch
couplings and |
Structured E/I sections |
|
|
Structured E/I sections |
|
One fixed-index sparse affine layer followed by ReLU. Every output is
active; the weights spread as evenly as possible over the outputs so
that weights plus biases equal the reference’s effective parameter
count. Needs |
Structured E/I sections |
|
One dense |
Structured E/I sections |
|
|
Structured E/I sections and |
model.core.target_active_parameters replaces the count taken from the
reference for sparse_active_matched_point. Matching parameter counts does
not equalize compute or activation memory.
A point_mlp control for the population network of
configs/examples/population_network_feedforward.yaml repeats the
population block under type: point_mlp:
model:
core:
type: point_mlp
population_network:
layers:
- name: layer0
readout_population: e
population_defaults:
ff_excitatory_synapses: 30
ff_inhibitory_synapses: 15
populations:
- {name: i, polarity: inhibitory, n_neurons: 32, branch_factors: [1]}
- {name: e, polarity: excitatory, n_neurons: 128, branch_factors: [4, 4]}
connections:
- {source: input, target: i}
- {source: input, target: e}
- {source: i, target: e}
A fixed-support control for a structured E/I core gives the reference sections and the support seed:
model:
core:
type: sparse_active_matched_point
architecture:
excitatory_layer_sizes: [64]
inhibitory_layer_sizes: [16]
excitatory_branch_factors: [2, 2]
inhibitory_branch_factors: [1]
connectivity:
ee_synapses_per_branch_per_layer: [16]
ei_synapses_per_branch_per_layer: [8]
ie_synapses_per_branch_per_layer: [4]
ii_synapses_per_branch_per_layer: [0]
sparsity:
indexed:
seed: 11
Dense and sparse affine MLPs
model.core.type: affine_mlp builds a conventional multilayer perceptron
with signed weights, in one of three modes, from
model.core.affine_mlp:
denseDense
nn.Linearlayers.fixedSparse layers with a fixed random support of
fanininputs per unit, stored as indices and weights (IndexedSparseLinear). Each layer’s support comes from its entry inconnectivity_seeds.topkLearned Top-K layers (
TopKLinear): every unit uses itsfaninweights of largest magnitude in each forward pass.
model:
encoder:
type: identity
params:
input_dim: 784
core:
type: affine_mlp
affine_mlp:
dimensions: [784, 256, 128]
mode: fixed
fanin: [64, 32]
connectivity_seeds: [101, 102]
dropout: 0.1
decoder:
type: MLP
params:
hidden_dims: []
output_dim: 10
The options of model.core.affine_mlp:
Option |
Default |
Meaning |
|---|---|---|
|
(required) |
Widths from input to output, at least two. The first must equal the encoder output size. |
|
|
|
|
|
Inputs per unit for each layer ( |
|
|
One support seed per layer, required by |
|
|
|
|
|
|
|
|
Slope of |
|
|
Apply the activation (and dropout) after the last layer too. Set it to
|
|
|
Dropout probability after each activation. |
|
|
Factor applied to the last layer’s initial weights. |
|
|
Execution settings of |
By default, dense mode constructs all layers, then draws every weight
with Kaiming-normal initialization (the ReLU gain, also for leaky_relu)
and sets the biases to zero. fixed layers draw normal weights with
standard deviation sqrt(2 / fanin). topk layers draw standard-normal
weights and rescale each unit’s weights so that its initially selected weights
have squared norm 2. All modes then multiply the last layer’s weights by
output_weight_scale. The weights come from the global PyTorch random
stream, which the builder does not reseed; connectivity_seeds fix only the
support of fixed layers.
For a conventional readout initialized by torch.nn.Linear, explicitly select
initialization: pytorch_default. This skips reinitialization, preserving the
constructors’ bias values and random draw order. With output_weight_scale: 1
it makes no further weight mutation; another scale still rescales the last
weight matrix but not its bias, which this policy leaves nonzero, so the
initial output is not scaled uniformly. Sparse modes reject this policy. For a Linear–ReLU–Linear
classifier, set dimensions to the three layer widths, activation: relu,
activate_output: false and dropout: null. This reuses the affine builder;
it does not add feature normalization or a dendritic computation.
The built model is a flat nn.Sequential (numeric checkpoint keys) that
also forwards the weight decay, weight boosting and rewiring steps of the
optimizer to its sparse layers. The same model can be built from Python:
from dendritic_modeling.config import AffineMLPConfig
from dendritic_modeling.networks.architectures.factory import get_architecture
config = AffineMLPConfig(
dimensions=[128, 64, 10],
mode="fixed",
fanin=[16, 8],
connectivity_seeds=[101, 10101],
activate_output=False,
)
model = get_architecture("affine_mlp", config, input_dim=128)
resolved = model.affine_mlp_config # the configuration as a dictionary
dendritic_modeling.networks.architectures.classical.mlp.build_affine_mlp(config)
returns the same model. To count the parameters of Top-K models,
dendritic_modeling.analysis.utils.model_resources.topk_parameter_inventory(model)
returns the trainable parameters, the Top-K candidate weights, the parameters
retained after Top-K selection and the buffer bytes; it counts parameters and
does not estimate memory or speed. topk layers can be converted to fixed
indexed layers for inference; see Compact checkpoints.
Recurrent baselines
gru,lstm,vanilla_rnn(aliasrnn)Standard PyTorch recurrent networks configured by
model.core.baseline_rnn:hidden_dim(default 64),num_layers,dropout,output_mode(last,meanoralltime steps),input_projection_dims(linear and ReLU layers before the recurrence),use_layernorm(layer normalization of the recurrence input and output),init_method(defaultororthogonalrecurrent weights),lstm_forget_biasand the GRU update-gate biasesgru_fast_update_bias,gru_slow_update_biasandgru_slow_fraction.heterogeneous_leak_ctrnn(aliasheterogeneous_ctrnn)A continuous-time point-neuron E/I network with sparse indexed contacts and leaky channels of several time constants per output neuron, configured by
model.core.heterogeneous_leak_ctrnn.tau_valueslists the time constants,owner_tau_patternsassigns them to the channels of each output neuron, and the*_contactsfields set the numbers of input, recurrent and readout contacts. The classHeterogeneousLeakCTRNNConfigindendritic_modeling.networks.architectures.recurrentchecks that these counts agree.legendre_memoryThe fixed linear memory of a Legendre delay network: each memory channel holds
memory_orderLegendre coefficients of its input over a sliding window of lengththeta(time stepdt). Sparse learned contacts write the input into the channels and read the coefficients out. It is configured bymodel.core.legendre_memory; the state transition is fixed, not trained.
The keys of these sections are checked when the model is built, and a
misspelled one raises an error that names the section and suggests the
closest key. input_dim is set from the encoder output and the cell of
baseline_rnn from the core type: a written input_dim that differs
from the encoder output, or a cell_type other than the core type’s cell,
raises.
Recurrent cores read sequences and train with
training.main.strategy: recurrent (see Training strategies). A GRU
baseline:
model:
core:
type: gru
baseline_rnn:
hidden_dim: 128
num_layers: 1
output_mode: last
A small heterogeneous-leak network, with two time constants per output neuron:
model:
core:
type: heterogeneous_leak_ctrnn
heterogeneous_leak_ctrnn:
input_projection_dims: [3]
n_excitatory_outputs: 2
n_inhibitory_outputs: 1
channels_per_output: 4
tau_values: [9.0, 3.0]
owner_tau_patterns:
- {owner_count: 3, tau_counts: [2, 2], excitatory_owner_count: 2}
input_contacts: 30
recurrent_excitatory_contacts: 18
recurrent_inhibitory_contacts: 9
readout_contacts: 4
topology_seed: 711
initialization_seed: 919
A Legendre memory with two memory channels of order 4 for sequences of three
input features. Each contact count lies between the larger and the product of
the two widths it connects: input_contacts between the input features and
the memory channels, readout_contacts between the
n_memory_channels * memory_order coefficients and output_dim.
dendritic-train sets model.encoder.params.input_dim from the data:
model:
encoder:
type: identity
params:
input_dim: 3
core:
type: legendre_memory
legendre_memory:
output_dim: 2
memory_order: 4
n_memory_channels: 2
theta: 8.0
input_contacts: 5
readout_contacts: 9
The recurrent E/I cores themselves are described in Recurrent structured cores, Unified E/I core and Population networks.
MLP building blocks
mlp (the decoder MLP) and effectivematchedparammlp are building
blocks that take flat parameters, not core types: model.core.type set to
one of them raises an error that names the alternatives. For a plain MLP core
use affine_mlp; for a matched control use point_mlp.
Core types lists the other names of this kind.
Feature normalization readouts
core.type: feature_readout composes the existing dense affine builder
with an explicit normalization of fixed feature maps. This is a conventional
control, not a biological E/I neuron. The optional core.feature_readout
section defaults to None in all other cores.
The input_shape excludes the batch and starts with channels. Inputs may
have this shape or be flattened; the latter are reshaped before normalization.
readout is an AffineMLPConfig.
Its first dimension must equal the product of input_shape. The readout is
constructed before the normalizer. A single unactivated Linear uses
readout.weight and readout.bias checkpoint keys; an MLP retains the
builder’s numeric module keys under readout. Training builds the readout
from the run’s global random stream (after any encoder draws), as it builds an
affine_mlp core, so normalization: none starts from the weights of the
equivalent affine_mlp control. An explicit core.initialization_seed
draws the readout from a CPU stream keyed by that seed and the core type.
This construction stays on CPU even under a CUDA default device, preserving
the caller’s CPU/CUDA random streams and default device. Move the resulting
model to the desired device afterwards, as the standard training setup does.
Four normalization choices are supported:
noneleaves features unchanged.layer_normuses PyTorch LayerNorm across the entire feature map.group_normuses PyTorch GroupNorm across contiguous channel groups and spatial sites.group_meandivides each feature bydivisor_offset + max(group_mean, 0). The mean includes channels within its group and every spatial site. This operation does not center features or use their absolute values. It is distinct from the existing L1 input normalizer and from the E/I shunting rule.
Grouped modes require groups to divide the channel count. Only group-mean
division accepts divisor_offset, which must remain finite and strictly
positive in the input dtype; values that overflow or round to zero raise before
division. LayerNorm and
GroupNorm use normalization_epsilon (default 1e-5) and
normalization_affine (default true). Irrelevant non-default options raise.
For group-mean division, forward and representation also accept an
explicit divisor_noise multiplier of shape [batch, groups]. It multiplies
the mean before clamping; it must be finite, nonnegative and match the input’s
dtype and device. No noise is sampled by the model.
configs/examples/feature_readout.yaml is a runnable example. It uses the
standard backpropagation trainer and existing synthetic data; it does not
reproduce an image-feature experiment. Other training strategies and sparse
readouts are currently rejected. Existing affine initialization defaults remain
unchanged; select readout.initialization: pytorch_default explicitly when
reproducing plain PyTorch Linear initialization, including its random biases.
Paired E/I routing controls
core.type: routed_ei builds a reduced rate-model control from
core.routed_ei. Each source has an excitatory and an inhibitory channel;
one assignment sends both channels to the same branch. Positive E and I
weights are normalized separately within each branch. The branches use
networks.functional.shunting_voltage or additive_voltage, followed by
a fixed affine rescaling and a learned affine decoder. The decoder has signed
weights: this control is not a biological population network or a multilevel
dendritic tree.
This composition reuses the library’s softplus weight transform, row-sum
division, current-integration arithmetic, architecture factory and standard
trainer.
It differs from pathway_router, which applies signed projections, and from
Top-K synapses, which select contacts within a branch rather than assign each
source to a destination. Both routers share the same soft-assignment utility.
configs/examples/routed_ei.yaml runs with dendritic-train and reuses
the existing branch_local_gain_load dataset. Validation uses the training
gain distribution; only the test set has the larger gain fluctuations. This
binary example is not a replay of a paper’s multiclass routing task. Other
datasets must supply finite nonnegative concatenated inputs [E, I] of
width 2 * features. Direct callers can supply the streams separately:
from dendritic_modeling.config import RoutedEIConfig
from dendritic_modeling.networks.architectures.excitation_inhibition.routed_ei import (
build_routed_ei,
)
config = RoutedEIConfig(features=8, branches=3, output_dim=2)
model = build_routed_ei(config)
logits = model(excitation, inhibition) # Each input: [batch, 8].
routing: learned uses a softmax over branch destinations. With
hard_routing: true (the default), training uses one-hot assignments with
the softmax surrogate derivative, computed as one_hot + soft -
soft.detach() like the reduced models this control replays (a selected
entry can differ from one in its last bit); evaluation uses exact one-hot
assignments. With hard_routing: false, both use soft assignments.
inferred_assignment() returns each source’s branch under hard routing and
resolves near-ties as the forward does. routing: fixed
requires a fixed_assignment list containing one branch index per feature.
Learned routing requires an empty list; non-list values are rejected before
initialization draws any random numbers. Fixed routing does not read
temperature, hard_routing or route_init_std, so
their non-default values raise. Assigning model.temperature re-validates
the configuration, so a temperature schedule applies to learned routing only.
At use, temperature must remain finite and strictly positive in the logits
dtype. The shared routing helper rejects nonfinite scaled logits before
softmax; this also protects statistics, inferred assignments and learned
pathway_router assignments. This check reads tensor values and may
synchronize the device. Valid assignments keep their original arithmetic.
Empty branches have zero drive; no implicit balancing or reassignment occurs.
inhibitory_support: global_mean is a shunting-only control that replaces
each branch’s inhibitory drive by the unweighted mean of the input I channels.
Shunting defaults to positive unit leak and epsilon 1e-8; a forward call
raises if either constant overflows in the model’s dtype or the leak rounds
to zero. Raw addition is
E - I; nondefault leak/epsilon values and global-mean support are rejected
because this comparator does not use them. The optional output transformation is
branch_scale * (voltage - branch_center) / branch_divisor; its default is
the identity. All three constants must remain finite in the voltage dtype,
with scale and divisor strictly positive after conversion. Signed centers
are supported. No gate, across-feature normalization or learned output scale is
implicit in this core. Set the outer model’s output_scale_mode: fixed
and use identity encoder/decoder modules to retain that contract. The control
computes in its parameter dtype: training.main.common.use_amp: true raises
when the configuration loads, and so does a forward call under
torch.autocast.
An explicit branch_activation: tanh applies
tanh(softplus(activation_slope_raw) * preactivation + activation_offset)
before the affine readout. The initial raw slope defaults to 0.5413249
(approximately unit softplus slope) and the offset to zero. These per-branch
values are fixed buffers unless trainable_activation: true is requested.
Initialization spreads and the optional gate’s initial constants must be
finite in the default dtype used to construct parameters. An invalid value
raises before initialization draws any random numbers. This check does not
rescale the specified values or change valid initialization streams.
This signed gate is not the neuronal ParametricTanh: that gate has a
unit-range output, an exponential slope and a voltage midpoint. Substituting
one for the other changes optimization even when an operating-point response
matches. Only the routed backpropagation control supports these new fields.
The default branch_activation: none registers no gate state and draws no
random numbers. Nondefault gate parameters with this setting raise rather
than being ignored. branch_response(E, I) explicitly returns currents,
preactivation, activation and the derivative of the gate with respect to its
preactivation. Ordinary forward calls do not calculate this diagnostic.
activate_branches(x) exposes the same configured gate to analysis-only
interventions without reimplementing its arithmetic. The original resolved
configuration and a strict state-dict round trip recover fixed or learned
gates, including their initial parameterization.
weight_normalization_floor: null divides each branch’s E and I weights by
their sum; an empty branch keeps zero weights. Unlike the robust p-norm
normalization of dendritic layers, the sum is also differentiated at the zero
entries of hard-routed rows, so the surrogate routing gradient keeps each
branch’s normalization (competition) term. An explicit positive floor selects
division by max(sum(weights), floor), the clamped-sum convention of earlier
reduced-model runs. The two conventions agree exactly, in values and
gradients, while every branch’s weight sum exceeds the floor; below it,
including at empty branches, the floor convention divides by the floor instead.
Replaying a clamped-sum model therefore requires its original floor. The floor
must remain finite and strictly positive in the weight dtype; values that
overflow or round to zero raise before division. The same check applies to
divide_rows_by_sum and normalize_weight_rows(..., sum_floor=...).
These sum-based paths also reject nonfinite row sums before division, including
overflow from finite weights, instead of silently zeroing a nonempty branch.
Finite sums retain the original normalization arithmetic and gradients.
The fixed-assignment and global-mean controls keep the unused route-logit or
I-weight parameters and their initialization draws, so all variants share one
checkpoint layout. These parameters do not require gradients: optimizers leave
them unchanged, and distributed data-parallel training does not wait for their
gradients. drives(E, I) returns the currents the forward integrates; under
global-mean inhibition its I current is the single shared mean, of shape
[batch, 1]. Store the resolved configuration alongside the state dict,
including any scheduled temperature changes.
Only ordinary backpropagation with the standard trainer is qualified.
LocalCA, feedback alignment and recurrent configurations raise during loading.
The affine readout prevents a biological_neuron: true declaration.
No production paper pin is changed merely by selecting this architecture.
Training builds the control from the run’s global random stream, as direct
construction after torch.manual_seed(model_seed) and any encoder draws
would. An explicit core.initialization_seed instead draws it from a CPU
stream keyed by that seed and the core type. Explicit-seed construction takes
place on CPU even when the caller’s default device is CUDA, and restores both
the caller’s device setting and random streams. Move the constructed core to
the desired device afterwards, as the standard training setup does. Without
an explicit seed, construction retains the caller’s default device and stream.
Routing regularization is opt-in under
training.main.regularization. routing_entropy_weight penalizes
mean source-assignment entropy; routing_balance_weight penalizes mean
squared deviation from equal branch mass. The regularization manager applies
these terms to modules exposing routing_statistics(), currently the paired
E/I and pathway routers. Positive weights request minimization of those
statistics. Fixed routers return no statistics, and requesting a routing
penalty on a model without a learned router raises. Both weights default to
zero, which performs no routing discovery or computation in existing runs.
routing_regularizer(entropy_weight=..., balance_weight=...) evaluates the
same term for a direct training loop; its weights have no defaults.
The configured-step benchmark supports no additional regularization and rejects
these penalties before model initialization, using the trainer’s regularization
manager to determine which terms are enabled.
Parameter-budget model families
dendritic_modeling.networks.benchmarks.parameter_budget builds whole
models (input encoding, hidden layers and a counted affine readout) for
comparisons at a fixed parameter budget, from Python. FAMILIES lists the
families:
denseDense hidden layers.
sparseFixed-support sparse hidden layers (
contacts_e + contacts_iinputs per unit).dendritic_signed,dendritic_additive,dendritic_shuntingConfigurableEINetworktrees with signed weights, or positive E/I conductances combined additively or by shunting.grouped_gatedA conventional control without dendritic dynamics: sparse gated subunits (as many per unit as the matching dendritic morphology has sites), group RMS normalization and a learned sum.
from dendritic_modeling.networks.benchmarks.parameter_budget import (
build_model,
match_parameter_budget,
model_report,
)
spec = {
"family": "dendritic_shunting",
"input_dim": 784,
"output_dim": 10,
"branch_factors": [2, 2],
"contacts_e": 8,
"contacts_i": 2,
}
model = build_model({**spec, "width": 64})
report = model_report(model) # counted parameters, storage, topology hash
matched_spec, matched_report = match_parameter_budget(spec, 100_000)
matched_model = build_model(matched_spec)
build_model seeds its own random stream (seed for parameters,
topology_seed, default seed, for the fixed support) and leaves the
caller’s generator untouched. Unknown spec keys raise. Further fields:
width (exposed hidden units), network_depth, activation (relu,
softplus, sigmoid, or the learned gates param_relu and
param_tanh, which need activation_init: {gain, threshold}),
input_encoding (signed_split, nonnegative or raw; positive
E/I families reject raw), initialization (fan_in, the default, or
legacy_xavier for Xavier-normal initialization), clip_contacts and
projection_backend. An optional readout selects a signed dense,
low_rank (rank) or indexed (topk) output allocation.
match_parameter_budget searches the integer width whose whole-model
parameter count is within tolerance (default 2 %) of the target and
raises when no width qualifies; it never pads with inert parameters. Matching
parameters does not equalize compute or activation memory.
Signed readouts
dendritic_modeling.networks.utils.readouts.make_signed_readout(input_dim,
output_dim, *, rank=None, topk=None, bias=False, init_std=0.02, seed=None,
projection_backend="eager") returns a dense nn.Linear (no budget), two
dense low-rank factors (rank), or an IndexedSparseLinear with fixed
support of topk inputs per output. rank and topk are mutually
exclusive, and indexed readouts take no bias.
Readout initialization
dendritic_modeling.training.readout_initialization.initialize_readout(model,
training_inputs, spec) optionally rescales an existing nn.Linear
model.readout before training:
{"mode": "preserve"}Leaves the model unchanged.
{"mode": "train_batch_rms", "target_rms": r, "examples": n}Multiplies the readout weights and bias by one factor so the logits of the first
ntraining inputs have root-mean-squarer. Only training inputs are used, never labels or validation data; the hidden state is verified unchanged by hash, and any failure restores the original state.
It returns a record of the calibration (initial and achieved RMS, scale, examples used, hashes) to store with the run: it changes the optimization starting point, not the model class or its parameter count.
Numerical utilities
Streaming covariance
dendritic_modeling.analysis.utils.streaming_covariance.StreamingCovariance(n_features)
merges the means and centered second moments of successive
(observations, features) batches in float64 without retaining
observations. covariance returns the sample covariance (n - 1
normalization) once two observations have been seen.
Passive branch and soma equilibrium
dendritic_modeling.networks.architectures.excitation_inhibition.dendritic.passive_equilibrium.passive_star_soma_voltage(excitation,
membrane_conductance, axial_conductance, soma_leak=1.0) returns the
steady-state soma voltage of independent passive stars: branches with input
current E, membrane load h > 0 and axial coupling g >= 0 around a
soma with leak to reversal zero. Each branch settles at
where \(s\) is the soma voltage, and the final dimension indexes the branches. The expression keeps a positive denominator, so it stays accurate at large \(g\), and it is differentiable.