kempnerforge.model.init

Weight initialization strategies for KempnerForge models.

Functions

init_weights(model, config)

Apply standard initialization to all parameters in a model.
kempnerforge.model.init.init_weights(model, config)[source]

Apply standard initialization to all parameters in a model.

Strategy (following GPT-2/Llama conventions):

  • Linear layers: normal(0, 0.02)

  • Embedding layers: normal(0, 0.02)

  • Residual output projections (o_proj, down_proj): scaled by 1/sqrt(2 * n_layers)

  • Cross-attention block residuals: zero-initialized (identity-at-init warm-start)

  • MoT per-modality residual projections: zero-initialized (identity-at-construction)

  • Norm layers and the sigmoid router’s own state: reset to their construction-time values, which a meta-device build does not preserve

Only the modules in _RESETS_OWN_STATE are reset, and only with deterministic fills: a blanket reset_parameters() would also run nn.Linear’s kaiming init, which draws from the RNG and changes every weight drawn after it for the same seed.

Parameters:
Return type:

None