Non-Negative Matrix Factorization (NMF)

Non-Negative Matrix Factorization (NMF) is a factorization method that decomposes a non-negative matrix into two lower-rank non-negative matrices. This technique is widely used in feature learning, dictionary learning, and dimensionality reduction, particularly in neural network activations.

The model consists of:

• Input Matrix (A): The pattern of activations in a neural network, shaped as (batch_size, n_features).
• Codes (Z): The representation of data in terms of discovered concepts, shaped as (batch_size, nb_concepts).
• Dictionary (D): A learned basis of concepts, shaped as (nb_concepts, n_features).

Basic Usage

from overcomplete import NMF

# define an NMF model with 10k concepts using the HALS solver
nmf = NMF(nb_concepts=10_000, solver='hals')

# fit the model to input activations A
Z, D = nmf.fit(A)

# encode new data
Z = nmf.encode(A)
# decode (reconstruct) data from codes
A_hat = nmf.decode(Z)

Solvers

The NMF module supports different optimization strategies: - HALS (Hierarchical Alternating Least Squares) - Efficient for large-scale problems. - MU (Multiplicative Updates) - Standard NMF update rule. - ANLS (Alternating Non-Negative Least Squares) - Robust least squares optimization. - PGD (Projected Gradient Descent) - Suitable for constrained optimization.

With HALS usually providing best reconstruction.

NMF

Torch NMF-based Dictionary Learning model.

__init__(self,
         nb_concepts,
         device='cpu',
         solver='hals',
         tol=0.0001,
         verbose=False,
         **kwargs)

Parameters

• nb_concepts : int

  • Number of components to learn.

• device : str, optional

  • Device to use for tensor computations, by default 'cpu'

• solver : str, optional

  • Optimization algorithm to use, by default 'hals'. Can be one of: - 'hals': Hierarchical Alternating Least Squares - 'mu': Multiplicative update rules - 'pgd': Projected Gradient Descent - 'anls': Alternating Non-negative Least Squares

• tol : float, optional

  • Tolerance value for the stopping criterion, by default 1e-4.

• verbose : bool, optional

  • Whether to display a progress bar, by default False.

init_random_z(self,
              A)

Initialize the codes Z using non negative random values.

Parameters

• A : torch.Tensor

  • Input tensor of shape (batch_size, n_features).

Return

• Z : torch.Tensor

  • Initialized codes tensor.

init_nndsvda(self,
             A)

Initialize the dictionary and the code using Sklearn NNDsvdA algorithm.

Parameters

• A : torch.Tensor

  • Input tensor of shape (batch_size, n_features).

Return

• Z : torch.Tensor

  • Initialized codes tensor.

• D : torch.Tensor

  • Initialized dictionary tensor.

get_dictionary(self)

Return the learned dictionary components from NMF.

Return

• torch.Tensor

  • Dictionary components.

encode(self,
       A,
       max_iter=300,
       tol=None)

Encode the input tensor (the activations) using NMF.

Parameters

• A : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, n_features).

• max_iter : int, optional

  • Maximum number of iterations, by default 100.

• tol : float, optional

  • Tolerance value for the stopping criterion, by default the value set in the constructor.

Return

• torch.Tensor

  • Encoded features (the codes).

sanitize_np_input(self,
                  x)

Ensure the input tensor is a numpy array of shape (batch_size, dims). Convert from pytorch tensor or DataLoader if necessary.

Parameters

• x : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, dims).

Return

• torch.Tensor

  • Sanitized input tensor.

fit(self,
    A,
    max_iter=500)

Fit the NMF model to the input data.

Parameters

• A : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, n_features).

• max_iter : int, optional

  • Maximum number of iterations, by default 500.

init_random_d(self,
              A)

Initialize the dictionary D using non negative random values.

Parameters

• A : torch.Tensor

  • Input tensor of shape (batch_size, n_features).

Return

• D : torch.Tensor

  • Initialized dictionary tensor.

sanitize_np_codes(self,
                  z)

Ensure the codes tensor (Z) is a numpy array of shape (batch_size, nb_concepts). Convert from pytorch tensor or DataLoader if necessary.

Parameters

• z : torch.Tensor

  • Encoded tensor (the codes) of shape (batch_size, nb_concepts).

Return

• torch.Tensor

  • Sanitized codes tensor.

decode(self,
       Z)

Decode the input tensor (the codes) using NMF.

Parameters

• Z : torch.Tensor

  • Encoded tensor (the codes) of shape (batch_size, nb_concepts).

Return

• torch.Tensor

  • Decoded output (the activations).