kempnerforge.model.norm

Normalization layers for KempnerForge models.

Functions

build_norm(norm_type, dim[, eps])

Build a normalization layer by name.

Classes

RMSNorm

Root Mean Square Layer Normalization (Llama-style).

class kempnerforge.model.norm.RMSNorm

Bases: Module

Root Mean Square Layer Normalization (Llama-style).

Simpler and faster than LayerNorm — no mean subtraction, no bias.

__init__(dim, eps=1e-05)

Parameters:

• dim (int)

• eps (float)

Return type:

None

forward(x)

Parameters:

x (torch.Tensor)

Return type:

torch.Tensor

kempnerforge.model.norm.build_norm(norm_type, dim, eps=1e-05)

Build a normalization layer by name.

Parameters:

• norm_type (str)

• dim (int)

• eps (float)

Return type:

torch.nn.Module