Datasets
data.dataset_name selects the dataset of a run (default cifar10).
data.processing holds the input options shared by all datasets, and
data.dataset_params.<dataset_name> holds the options of the selected
dataset. experiment.train_valid_split and the experiment test-split
options decide how the data are divided.
experiment:
train_valid_split: 0.8
data:
dataset_name: mnist
base_dir: "" # empty: DENDRITIC_DATA_DIR or the user cache
processing:
flatten: true
normalize: false
dataset_params: {}
Selecting a dataset
An unknown data.dataset_name raises when the datasets are built, with the
closest known name and the full list of names. The
Dataset catalogue below lists every name the loaders accept.
data.dataset_params maps dataset names to option blocks. Only the block
named by data.dataset_name is read; blocks for other datasets are ignored,
so one file can keep the options of several datasets. A key in the selected
block that its loader does not read raises, with a close match when there is
one. These keys are checked when the dataset is built (at the start of
training, or with dendritic-train config.yaml --validate-only), not when
the configuration file loads. Annotations for people or other tools belong in
the top-level metadata: section.
A flatten or normalize key in the selected dataset_params block
takes precedence over data.processing. Generated datasets draw their
samples from experiment.dataset_seed unless the block sets seed, and
random train/validation partitions use experiment.split_seed; both fall
back to experiment.seed (see Random seeds and reproducibility flags).
Dataset catalogue
Every value of data.dataset_name that the loaders accept. A name ending in
_positive is the positive-input variant of the task before it (see
Sequence datasets).
|
Family |
Content |
|---|---|---|
|
MNIST digits, 28x28 grayscale, 10 classes. |
|
|
Fashion-MNIST, 28x28 grayscale, 10 classes. |
|
|
CIFAR-10, 32x32 color, 10 classes. |
|
|
CIFAR-100, 32x32 color, 100 classes. |
|
|
ILSVRC-2012 classification from a local copy, 1000 classes. |
|
|
Two MNIST digits; label |
|
|
Two CIFAR-10 images; label |
|
|
Two MNIST digits and a context cue; label |
|
|
Two CIFAR-10 images and a context cue; label |
|
|
MNIST or CIFAR-10 intensities as Poisson rates, with optional multiplicative gain. |
|
|
Gaussian E and I features with set within- and cross-group correlations; classes differ in E versus I activity. |
|
|
Two classes separated by a hyperplane, correlated E/I features. |
|
|
Two classes with a quadratic, XOR, multiplicative or radial boundary. |
|
|
XOR-like task where the class depends on products of input pairs. |
|
|
Feature blocks with different correlation structures. |
|
|
Class information spread evenly, concentrated, or graded across features. |
|
|
A few highly informative features among weakly informative ones. |
|
|
Low-dimensional XOR, quadratic or multiplicative structure. |
|
|
Different noise levels on the E and I features. |
|
|
Mixed-selectivity or ring population code projected onto E and I inputs. |
|
|
Two input streams and a context cue that selects the relevant stream; stream gains shift between training and test. |
|
|
Positive E and I streams under group-wise multiplicative gain and optional inhibitory load. |
|
|
Positive E and I streams under nested multiscale gain fields. |
|
|
MNIST read pixel by pixel or row by row. |
|
|
Reproduce a symbol sequence after a delay. |
|
|
Sum the two marked values of a long sequence (regression). |
|
|
Delayed match-to-sample with a variable delay. |
|
|
Delayed match-to-sample with interleaved distractors and a fast secondary task, in counterfactual pairs. |
|
|
A context cue decides which block type to count over a sequence of noisy blocks. |
|
|
Integrate the evidence of the channel a context cue selects. |
|
|
Another name for the task above, with the same defaults. |
|
|
Per-step decisions while the relevant channel switches in long blocks. |
|
|
Delayed match-to-sample whose delay range can differ between splits. |
|
|
Reproduce the Ready-Set interval with a Go time. |
|
|
Two competing Ready-Set streams; a context cue selects the one to reproduce. |
|
|
Interval reproduction with a prior cue; targets are posterior- predictive Go-time distributions. |
|
|
Interval reproduction with a prior cue and one sampled Go step per trial. |
|
|
Oculomotor delayed response: report a cued angle after a delay. |
|
|
Compare two vibrotactile-like frequencies separated by a delay. |
|
|
Cued delayed comparison under global and pathway-specific gain. |
|
|
Classify the dominant frequency of a noisy mixture of sinusoids. |
|
|
Next-step forecasting of a quasi-periodic signal (regression). |
|
|
Lorenz-attractor next-state prediction (regression); two names for one loader. |
|
|
Partially observed switching linear dynamical system (regression). |
|
|
Mouse V1 responses to drifting gratings from a local copy of the recordings; orientation classification. |
|
|
EMNIST Balanced with context-shifted labels and per-presentation gain
and noise corruption; needs |
Training, validation and test splits
Every loader returns three splits. The test split is the dataset’s own test set: the MNIST and CIFAR test sets, the official ImageNet validation set, or a separately generated test set.
Training and validation
experiment.train_valid_split (default 0.8) is the fraction of the
training data kept for training; the rest forms the validation split, which
selects the best checkpoint (training.main.common.load_best_state_dict)
and drives early stopping. The partition is random, seeded by
experiment.split_seed. It applies to:
the vision, paired arithmetic, Poisson and ImageNet datasets (the paired datasets split their base images before pairing them);
sequence datasets that generate one training set and one test set, which is every sequence dataset without explicit splits (below). The counterfactually paired datasets (
contextual_ready_set_goandbayesian_ready_set_go) keep both members of a pair in the same split;stringer_v1, whose fixed test split is 10% of the trials.
It does not apply to:
the synthetic datasets. The first ten split one generated set 60/20/20 into training, validation and test;
contextual_stream_gain_shift,branch_local_gain_loadandhierarchical_gain_loadgenerate each split separately (see Synthetic datasets);sequence datasets with explicit splits;
registered datasets such as
contextual_remapping, which define their own splits.
The value must lie in [0, 1]; other values raise when the configuration
loads, and 0 raises when the datasets are built for every dataset except
registered ones. At 1.0 the datasets above have an empty validation
split. A run then stops during setup when
training.main.common.load_best_state_dict or early_stopping is true
(both are by default), because both select on the validation loss; with both
false, the validation loss is recorded as NaN.
Explicit splits for sequence tasks
Several sequence builders accept train, valid and test mappings
inside their dataset_params block. Each mapping overrides task parameters
for one split, and experiment.train_valid_split is then not used. This is
how a test split can use delays that the training split never sees:
data:
dataset_name: timescale_generalization
dataset_params:
timescale_generalization:
n_samples: 20000
train: {min_delay: 10, max_delay: 50}
valid: {min_delay: 10, max_delay: 50}
test: {min_delay: 60, max_delay: 100}
A split without a mapping uses the common values. n_samples in a mapping
sets the size of that split; by default the training split has n_samples
examples and the validation and test splits have
max(n_samples // 5, 2000).
Datasets |
Explicit splits |
|---|---|
|
Optional. Any task parameter may differ between splits. |
|
Optional. Each split draws its own seed ( |
|
Always. The splits are generated separately, with seeds |
Sealed test split
These experiment options protect the test split of a confirmatory run:
max_test_samples(defaultnull)Upper bound on the size of the built test split. A larger test split raises; nothing is subsampled. It applies whenever it is set.
sealed_test(defaultfalse)Seals the test split. It requires
allow_test_data: falseandmax_test_samples, and training then refuses to start while any analysis is enabled, since analyzers read the test split.allow_test_data(defaulttrue)falseis part of the seal and is accepted only withsealed_test: true.record_dataset_fingerprints(defaultfalse)Writes
dataset_fingerprints.jsonto the run directory: content hashes (SHA-256) of the tensors and stored attributes of the training, validation and test datasets, and one combined hash. For file-backed datasets such as ImageNet the hash covers the stored file list, not the image bytes.
A sealed run on a generated dataset whose test split has 400 examples:
experiment:
sealed_test: true
allow_test_data: false
max_test_samples: 400
record_dataset_fingerprints: true
data:
dataset_name: correlated_gaussian
dataset_params:
correlated_gaussian:
n_samples: 1000
The check that rejects enabled analyses is part of training:
dendritic-analyze-checkpoint runs the analyses it is given on a saved
checkpoint even when its training configuration seals the test split (see
Analyzing saved checkpoints).
Where data are stored
The data root is data.base_dir when it is set; otherwise
$DENDRITIC_DATA_DIR; otherwise $XDG_CACHE_HOME/dendritic_modeling
(~/.cache/dendritic_modeling when XDG_CACHE_HOME is unset), so runs
share one copy. dendritic-train --data_dir <path> overrides
data.base_dir. Each run logs the data root and records
DENDRITIC_DATA_DIR in runtime_environment.json.
Torchvision datasets are downloaded on first use into subdirectories of the root:
<root>/datasets/mnist: MNIST, also used bysequential_mnist, the MNIST paired datasets and the Poisson generator on MNIST;<root>/datasets/fashion_mnist: Fashion-MNIST;<root>/datasets/cifar: CIFAR-10 and CIFAR-100, also used by the CIFAR-10 paired datasets and the Poisson generator on CIFAR-10.
Processes that load the same directory at once take turns through a
.download.lock file in it. A missing or corrupted copy is deleted and
downloaded again.
Three datasets read local files and never download by default: ImageNet (see
ImageNet), stringer_v1 (see Neural recordings: stringer_v1)
and contextual_remapping, which downloads EMNIST only with
download: true (see Registered datasets).
Input processing
data.processing has three options. A written true that the selected
loader cannot apply raises instead of being ignored.
flatten(default: unset, which loaders treat astrue)Return inputs as vectors. Vision, paired and Poisson datasets flatten when it is unset; ImageNet then raises, so ImageNet runs set
flatten: false. Synthetic datasets are already flat and accepttrue. Sequence and registered datasets keep their own layout and raise ontrue.normalize(default: unset, which loaders treat asfalse)Dataset-specific input scaling. For the vision, paired and ImageNet datasets,
truesubtracts the per-channel mean and divides by the standard deviation (torchvisionNormalizewith the dataset’s statistics).sequential_mnistdivides pixel values by 255 andstringer_v1z-scores every neuron, both unlessnormalize: false. The synthetic, Poisson, other sequence andcontextual_remappingdatasets have no input scaling and raise ontrue.label_noise_rate(float in[0, 1], default0.0)Fraction of training labels replaced by a class drawn uniformly from all classes, including the original one, so about
label_noise_rate * (C - 1) / Cof the labels change forCclasses. Only the training split changes. The draws useexperiment.dataset_seed. Onlymnist,fashion_mnist,cifar10andcifar100apply it; a positive rate for any other dataset raises.
data:
dataset_name: cifar10
processing:
flatten: true
normalize: true
label_noise_rate: 0.2
Shunting models with positive weight transforms expect non-negative input to
their first shunting layer. normalize: true makes the inputs signed: keep
normalize: false, or rectify the transfer layer of a structured E/I core
with model.core.transfer.output_activation set to relu, softplus
or sigmoid.
Vision datasets
mnist, fashion_mnist, cifar10 and cifar100 load the
torchvision datasets into memory as tensors with values in [0, 1]
(before normalize). With the default flatten the inputs are vectors of
784 (MNIST, Fashion-MNIST) or 3072 (CIFAR) values; with flatten: false
they keep the [channels, height, width] shape. They read no
dataset_params keys.
Paired arithmetic datasets
Each split is built from the matching split of the base dataset: the images
are permuted and the first half is paired with the second half, so one pass
gives n // 2 pairs. shuffle_iterations repeats the pairing with
further permutations and multiplies the split size accordingly.
mnist_modulo10Concatenates the two images along the last axis (a 1568-value vector by default) and labels the pair
(a + b) mod 10. Options:shuffle_iterations: 2,pair_seed: 0.cifar10_modulo10Labels a pair of CIFAR-10 images
(a + b) mod 10. With the defaultflattenthe input is the two flattened images; withflatten: falsethe images sit side by side. Options:shuffle_iterations: 1,pair_seed: 0.double_mnist_contextual_mod10,double_cifar10_contextual_mod10Draw a random context for every pair (seeded by
context_seed). Context 0 labels the pair(a + b) mod 10and context 1(a - b) mod 10. The context is appended as a two-value one-hot vector, or as two constant image planes withflatten: false. Options:shuffle_iterations: 1,pair_seed: 0,context_seed: 123.
data:
dataset_name: double_mnist_contextual_mod10
dataset_params:
double_mnist_contextual_mod10:
shuffle_iterations: 2
pair_seed: 0
context_seed: 123
Poisson generator
poisson_generator treats the pixel intensities of a base dataset as rates.
Every access draws fresh noise, so each epoch and each evaluation sees
different samples:
where T = stimulus_duration and g is the gain. Options of
data.dataset_params.poisson_generator:
base_dataset(defaultmnist)mnistorcifar10. Any other name raises. The base images are not normalized.poisson_sampling(defaulttrue)falsereturnsg xwithout noise.stimulus_duration(default1.0)T: longer stimuli give more counts and relatively less noise.multiplicative_gain(defaulttrue)Apply the gain
g. Withfalsethe gain options are not used.fixed_gain_factor(defaultnull)One gain for every input. It takes precedence over the bounds below.
max_gain_factor(defaultnull)Draw
gfrom[1, max_gain_factor]; it must be at least 1.max_gain_tau_ratio(defaultnull)Sets the upper bound to
max_gain_tau_ratio * stimulus_durationwhenmax_gain_factoris not set.gain_sampling(defaultlog_uniform)log_uniformdrawslog guniformly;linear_uniformdrawsguniformly.uniform_gain(defaulttrue)truedraws one gain per sample;falsedraws one gain per input element and needs a fixed gain or an upper bound.
With the defaults no bound is set and the gain is 1, so only Poisson noise is
added. Of the processing options only flatten applies; normalize: true
and label noise raise.
data:
dataset_name: poisson_generator
dataset_params:
poisson_generator:
base_dataset: mnist
max_gain_factor: 10.0
stimulus_duration: 1.0
Synthetic datasets
Synthetic datasets are generated in memory from experiment.dataset_seed
(or the block’s seed). Their inputs are flat vectors.
The first ten datasets below generate
n_samplesexamples per class, with inputs in[0, 1], shuffle them and split the set 60/20/20 into training, validation and test.nonlinear_interaction,nonlinear_separableandpopulation_codeassign classes after sampling and drop examples of the larger class to balance them, so their total can be smaller thann_samplestimes the number of classes. Inputs combineneexcitatory-designated andniinhibitory-designated features, andinput_dimmust equalne + niwhere both appear.contextual_stream_gain_shift,branch_local_gain_loadandhierarchical_gain_loadgenerate the training, validation and test splits separately, with aboutn_samplesexamples each. Their test split is drawn from a shifted gain distribution, and so is their validation split unlessvalid_split_mode: train: checkpoint selection and early stopping then see test-distribution data. Leavingvalid_split_modeunset logs a warning; set it explicitly.
The options and their defaults (every block also accepts seed):
Dataset |
Options (defaults) |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
contextual_stream_gain_shift inputs are [stream A | stream B | context]
(2 * stream_dim + 2 values in [0, 1]); the gains change contrast around
0.5. The two gain-load datasets produce positive [E | I] streams
(2 * stream_dim values) for a transfer layer with independent pathways.
In branch_local_gain_load a mean-one log-normal gain, with log standard
deviation train_gain_sigma in the training split and test_gain_sigma
in the test split, multiplies groups of neighbouring features of both
streams. With the defaults both are 0.8 on the same features, so there is no
train/test shift; set different values, or gain_alignment_alpha below 1 to
move part of the test gain off the class-signal features.
hierarchical_gain_load shifts by default (0.4 in training, 1.4 in test).
The generator docstrings in dendritic_modeling.datasets.synthetic_datasets
define every option.
data:
dataset_name: correlated_gaussian
dataset_params:
correlated_gaussian:
n_samples: 1000
input_dim: 100
n_classes: 2
rho_ee: 0.3
rho_ii: 0.3
rho_ei: 0.0
Sequence datasets
Sequence datasets return inputs of shape [time, features] for recurrent
cores (a recurrent population_network or one of the recurrent cores in
Core types). The target is a class label, a per-step label sequence, a
distribution over time steps, or a regression target, as listed per dataset.
Tasks with variable trial lengths also return each trial’s length. Regression
tasks (adding_problem, lorenz, lorenz_sequence,
multi_sine_forecast, switching_lds) need model.task: regression.
Shared conventions:
Sizes.
n_samplesis the size of the training data before the validation split. The test split hasmax(n_samples // 5, minimum)examples, with a minimum of 1000 forcopy_task,adding_problem,variable_delay_dmsand the signal tasks and 2000 for the others.sequential_mnistandstringer_v1use the recorded data instead.Seeds.
seeddefaults toexperiment.dataset_seed. Tasks that build their test split from a second seed default it toseed + 1(test_seedwhere listed).Positive encoding. A
_positivename selects inputs that are never negative, for shunting models with positive conductances. Settingpositive_input_encoding: trueunder the base name selects the same encoding.Processing.
flattenandnormalizeraise whentrue(exceptnormalizeforsequential_mnistandstringer_v1), and label noise raises.
data:
dataset_name: context_dependent_decision
dataset_params:
context_dependent_decision:
n_samples: 20000
signal_dim: 8
noise_std: 1.0
variable_integration: true
min_integration: 20
max_integration: 80
Memory tasks
Dataset |
Input and target |
Options (defaults) |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
stimulus, delay, probe; binary match label |
|
|
|
|
Decision tasks
Dataset |
Input and target |
Options (defaults) |
|---|---|---|
|
context cue, then |
|
|
context cue and two evidence channels ( |
|
|
|
|
|
stimulus, delay, probe; binary match label |
|
In context_dependent_decision, variable_integration: true draws each
trial’s integration length from [min_integration, max_integration] and
ignores integration_duration (with a warning); false uses
integration_duration. The delay after integration is set by
delay_duration or by min_delay/max_delay. The option’s former
name variable_delay still loads, with a warning. For a harder task, raise
conflict_fraction, noise_std or the distractor_burst_* options;
the _hard names build the same task with the same defaults.
Interval timing tasks
In the Ready-Set-Go tasks a Ready pulse and a Set pulse are separated by a
sampled interval, and the model reproduces the interval after Set. The target
is the Go time, for training.main.common.loss_function: temporal_event_ce:
a normalized distribution over the time steps (ready_set_go,
contextual_ready_set_go, bayesian_ready_set_go) or one Go step
(factual_bayesian_ready_set_go). The sequence length is fixed for all
trials; when sequence_duration is unset it is the shortest length that
fits every trial. set_min_step and set_max_step bound the Set time and
default to values derived from the intervals.
Dataset |
Input and target |
Options (defaults) |
|---|---|---|
|
3 channels (Ready, Set, distractor pulses); Go-time distribution |
|
|
7 channels (two Ready-Set streams, two context channels, distractors); Go-time distribution; counterfactual pairs |
the |
|
5 channels (Ready, Set, short- and long-prior cues, distractors);
posterior-predictive Go-time distribution; counterfactual pairs
( |
|
|
5 channels as above; one Go step per trial |
the |
Neuroscience-inspired tasks
All three accept explicit splits (see Training, validation and test splits).
Dataset |
Input and target |
Options (defaults) |
|---|---|---|
|
fixation, angular cue, delay, go; class of |
|
|
first frequency, delay, second frequency, response; binary label (second frequency higher) |
|
|
pathway cue, two stimuli on every pathway separated by a delay; binary label (the cued pathway’s second value is larger) |
|
Signal and dynamics tasks
Dataset |
Input and target |
Options (defaults) |
|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
The dataset classes in dendritic_modeling.datasets.sequence document each
option. A dataset whose builder is not listed here, added with
register_sequence_dataset_builder, checks its own options.
Neural recordings: stringer_v1
stringer_v1 classifies the orientation of drifting gratings from mouse V1
population responses (Stringer et al., 2019, Science 364). Each example is a
static vector of n_neurons responses, not a sequence, so it suits
feedforward cores. The data are not downloaded: data_dir (default: the data
root) must contain the response file grating_file (default
gratings_drifting_GT3_2019_04_05_1.npy) and all_running.npy;
session_id selects another session by name and then also needs
database.npy.
n_neurons(default500)Number of neurons kept, ranked by response variance.
n_classes(default8)Number of orientation bins over
[0, 2*pi).normalize(defaulttrue)Z-score every neuron.
neuron_selection_scope(defaultall_trials)all_trialsranks neurons on all trials (transductive);train_splitranks them on the training trials only and reuses that ranking for the test trials.
The test split is a fixed 10% of the trials drawn with the dataset seed, and
the validation split comes from the remaining trials through
experiment.train_valid_split. The block also accepts running-state
options (running_state, running_quantile, train_running_state,
test_running_state, state_holdout_fraction,
running_residualize, running_residualize_clip), gain augmentation
(gain_group_count, gain_grouping, global_gain_min/max,
group_gain_min/max, gain_seed) and Poisson resampling
(poisson_sampling, poisson_rate_scale). The gain, Poisson and
residualization options also take train_ and test_ prefixes to differ
between the training and test trials. Setting train_running_state and
test_running_state to disjoint states (low and high) trains on one
running state and tests on the other; see StringerV1Dataset in
dendritic_modeling.datasets.sequence.neuro_tasks.
Registered datasets
contextual_remapping
The context-cued EMNIST Balanced task (47 classes). Each presentation shows an
image with a random context c of num_contexts; the label is
(label + c) mod 47 and the context is appended as a one-hot cue scaled by
context_scale. The image (not the cue) is corrupted by a log-uniform gain
in [gain_min, gain_max] and brightness-dependent Gaussian noise whose
divisor is log-uniform in [noise_divisor_min, noise_divisor_max].
It requires training.main.strategy: presentation_stream: the training
split serves freshly corrupted presentations, and indexing it returns clean
context-0 items that only fix the input shape. It defines its own splits:
pool_per_class training images per class are held out, the first
validation_per_class of them form a fixed corrupted validation set, and
the test split is the corrupted official EMNIST test set. EMNIST is read from
<root>/raw (root defaults to the data root) and downloaded there only
with download: true. Processing options raise.
Options and defaults: root: "", download: false, upright: true,
peak_normalize: true, pool_per_class: 200,
validation_per_class: 50, partition_seed: 31003, num_contexts: 32,
context_scale: 1.0, gain_min: 0.05, gain_max: 20.0,
noise_divisor_min: 1.0, noise_divisor_max: 4.0,
validation_seed: 78019, test_seed: 99173, generator_device: cpu
(cuda draws the evaluation corruptions on a GPU). Unknown keys raise.
training:
main:
strategy: presentation_stream
data:
dataset_name: contextual_remapping
dataset_params:
contextual_remapping:
download: true
num_contexts: 32
The model configurations that use this task are listed in Example map, and Conductance options describes the conductance options they use.
Adding a dataset
dendritic_modeling.datasets.registry.register_dataset(name, builder, ...)
adds a dataset selectable by data.dataset_name. The builder receives the
task configuration (attributes dataset, data_path,
train_valid_split and parameters, the merged processing options,
dataset parameters and seeds) and returns (train, valid, test). Keyword
options declare a required_strategy and the processing options the
builder applies (others then raise). Registered names are looked up before the
built-in loaders, so register a name that no built-in dataset uses.
dendritic_modeling.datasets.sequence.register_sequence_dataset_builder
adds a sequence builder that the shared sequence factory splits. See
Extension registries.
ImageNet
imagenet loads ILSVRC-2012 from a local copy; it is never downloaded, and
a missing copy raises with the resolved paths (there is no synthetic
fallback). Point data.base_dir at the ImageNet root:
data:
dataset_name: imagenet
base_dir: "/path/to/imagenet"
processing:
flatten: false
normalize: true
dataset_params:
imagenet:
transform_preset: pretrained
loader_backend: auto
Two layouts are recognized in the data root, and otherwise in
<root>/datasets/imagenet:
a torchvision ImageNet root: the devkit (
meta.bin,ILSVRC2012_devkit_t12.tar.gzorILSVRC2012_devkit_t12/) with training (train/orILSVRC2012_img_train/) and validation (val/orILSVRC2012_img_val/) images;class directories:
train/<class>/(orILSVRC2012_img_train/<class>/) andval/<class>/, loaded with torchvisionImageFolder.
Options of data.dataset_params.imagenet:
loader_backend(defaultauto)autouses a torchvision root when one is found and class directories otherwise;torchvisionforcestorchvision.datasets.ImageNet;imagefolderforces class directories and skips the devkit metadata.transform_preset(defaultlegacy_resize)legacy_resizeresizes every image to 224x224.pretrainedusesRandomResizedCrop(224)and horizontal flips for training andResize(256)thenCenterCrop(224)for evaluation; use it for torchvision pretrained backbones.train_samples_per_class,val_samples_per_class(defaultnull)Keep a seeded random subset of at most this many images per class.
train_samples_per_classlimits the training images before the validation split is drawn from them;val_samples_per_classlimits the test split.dendritic-train --train_samples_per_class N --val_samples_per_class Msets them from the command line for short test runs.allow_flatten_materialize(defaultfalse)ImageNet requires
processing.flatten: false; an unsetflattencounts astrueand raises.allow_flatten_materialize: trueinstead materializes flattened tensors in memory, keeping at most the first 50,000 examples of each split. It is meant for debugging only.
The validation split is the random 1 - train_valid_split fraction of the
training images, seeded by experiment.split_seed and loaded with the
evaluation transform. The test split is the official validation set.
normalize: true applies the ImageNet channel mean and standard deviation.
Two environment variables help with large copies on shared filesystems:
IMAGENET_LOCAL_DIRWhen it names an existing directory, ImageNet is loaded from there instead of the data root, for example from a copy on local disk.
DENDRITIC_IMAGEFOLDER_CACHEA directory for a cached
ImageFolderindex (class names and file list). The first run scans the class directories and writes the index; later runs read it instead of scanning about 1.28 million files.
ImageNet protocol loaders
dendritic_modeling.datasets.imagenet_protocol builds single ImageNet
splits of a run without loading the others, for example to evaluate another
model on exactly the images a run trained, selected and tested on. Both
functions take the run’s configuration (as returned by
dendritic_modeling.config.load_config), require
data.dataset_name: imagenet and processing.flatten: false, and read
the ImageNet location, loader_backend, transform_preset and
normalize from it. They find the data where training does:
data.base_dir, else $DENDRITIC_DATA_DIR, else the cache directory.
build_protocol_train_partition(config)Returns
(train, valid, info): the training subset with the training transform and the disjoint validation subset with the evaluation transform, partitioned exactly as in training (train_valid_splitandexperiment.split_seed, which falls back toexperiment.seed). It raises whentrain_samples_per_classis set.build_protocol_test_dataset(config)Returns
(test, info): the official validation set with the evaluation transform. It raises whenval_samples_per_classis set.
info records the ImageNet directory, backend, transform preset and
transforms, normalize, and the split sizes (and, for the training
partition, train_valid_split and the split seed).
Loading during training
The DataLoader settings of training are fields of training.main.common
(num_workers, pin_memory, persistent_workers, prefetch_factor,
in_order), listed under “Training loop options” in
Training; analyses use analysis.runtime instead (see
Data iteration: analysis.runtime). Except under FSDP, training records the
resolved settings under dataloader_tuning in training_summary.json.
Left unset, they depend on the dataset type:
In-memory
TensorDatasetdata (the vision and synthetic datasets, also insideSubsetorConcatDataset) load in the main process.Other datasets, including the package’s generated sequence datasets, the Poisson generator and the paired datasets other than
mnist_modulo10, use worker processes. Setnum_workers: 0to load them in the main process. From the second epoch on, the shuffle order and the per-sample random draws differ between main-process, persistent-worker and non-persistent-worker loading, so keep these settings fixed when runs must match.ImageNet and other torchvision image-folder datasets use workers with
prefetch_factor: 4andin_order: false.