kempnerforge.config.optimizer

Optimizer configuration.

Functions

checked_lr_multipliers(multipliers)

Return multipliers as a {module: multiplier} dict of floats.

Classes

OptimizerConfig

Optimizer settings.

kempnerforge.config.optimizer.checked_lr_multipliers(multipliers)

Return multipliers as a {module: multiplier} dict of floats.

Raises:

ValueError – multipliers is not a dict, a key is not a non-empty string, or a value is not a positive finite number.

Parameters:

multipliers (object)

Return type:

dict[str, float]

class kempnerforge.config.optimizer.OptimizerConfig

Bases: object

Optimizer settings.

name: str = 'adamw'

lr: float = 0.0003

weight_decay: float = 0.1

betas: tuple[float, float] = (0.9, 0.95)

eps: float = 1e-08

fused: bool = True

muon_momentum: float = 0.95

muon_ns_steps: int = 5

muon_adam_lr: float | None = None

muon_nesterov: bool = False

muon_adjust_lr: str = 'original'

muon_orthogonalize: str = 'local'

muon_router: bool = False

schedule_free_warmup_steps: int = 0

weight_decay_scope: str = 'exclude_norms_and_biases'

lr_multipliers: dict[str, float]

__init__(name='adamw', lr=0.0003, weight_decay=0.1, betas=(0.9, 0.95), eps=1e-08, fused=True, muon_momentum=0.95, muon_ns_steps=5, muon_adam_lr=None, muon_nesterov=False, muon_adjust_lr='original', muon_orthogonalize='local', muon_router=False, schedule_free_warmup_steps=0, weight_decay_scope='exclude_norms_and_biases', lr_multipliers=<factory>)

Parameters:

• name (str)

• lr (float)

• weight_decay (float)

• betas (tuple[float, float])

• eps (float)

• fused (bool)

• muon_momentum (float)

• muon_ns_steps (int)

• muon_adam_lr (float | None)

• muon_nesterov (bool)

• muon_adjust_lr (str)

• muon_orthogonalize (str)

• muon_router (bool)

• schedule_free_warmup_steps (int)

• weight_decay_scope (str)

• lr_multipliers (dict[str, float])

Return type:

None