Semi-Nonnegative Matrix Factorization (Semi-NMF)

Semi-Nonnegative Matrix Factorization (Semi-NMF) is a variant of NMF that relaxes the constraint on the code matrix (Z), allowing negative values while keeping the dictionary (D) nonnegative. The model consists of:

• Input Matrix (A): The pattern of activations in a neural network, shaped as (batch_size, n_features).
• Codes (Z): The representation of data in terms of discovered concepts, shaped as (batch_size, nb_concepts), and can take negative values.
• Dictionary (D): A learned basis of concepts, shaped as (nb_concepts, n_features), constrained to be nonnegative.

Basic Usage

from overcomplete import SemiNMF

# define a Semi-NMF model with 10k concepts using
# the multiplicative update solver
semi_nmf = SemiNMF(nb_concepts=10_000, solver='mu')

# fit the model to input activations A
Z, D = semi_nmf.fit(A)

# encode new data
Z = semi_nmf.encode(A)
# decode (reconstruct) data from codes
A_hat = semi_nmf.decode(Z)

Solvers

The Semi-NMF module supports different optimization strategies: - MU (Multiplicative Updates) - Standard Semi-NMF update rule. - PGD (Projected Gradient Descent) - Allows for sparsity control via L1 penalty on Z.

For further details, we encourage you to check the original references ¹.

SemiNMF

Torch Semi-NMF-based Dictionary Learning model.

__init__(self,
         nb_concepts,
         solver='mu',
         device='cpu',
         tol=0.0001,
         l1_penalty=0.0,
         verbose=False,
         **kwargs)

Parameters

• nb_concepts : int

  • Number of components to learn.

• solver : str, optional

  • Solver to use, by default 'mu'. Possible values are 'mu' (multiplicative update) and 'pgd' (projected gradient descent), 'pgd' allows for sparsity penalty.

• device : str, optional

  • Device to use for tensor computations, by default 'cpu'

• tol : float, optional

  • Tolerance value for the stopping criterion, by default 1e-4.

• l1_penalty : float, optional

  • L1 penalty for the sparsity constraint, by default 0.0. Only used with 'pgd' solver.

• verbose : bool, optional

  • Whether to print the status of the optimization, by default False.

init_random_z(self,
              A)

Initialize the codes Z using random values (can be negative).

Parameters

• A : torch.Tensor

  • Input tensor of shape (batch_size, n_features).

Return

• Z : torch.Tensor

  • Initialized codes tensor.

get_dictionary(self)

Return the learned dictionary components from Semi-NMF.

Return

• torch.Tensor

  • Dictionary components.

encode(self,
       A,
       max_iter=300,
       tol=None)

Encode the input tensor (the activations) using Semi-NMF.

Parameters

• A : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, n_features).

• max_iter : int, optional

  • Maximum number of iterations, by default 300.

• tol : float, optional

  • Tolerance value for the stopping criterion, by default the value set in the constructor.

Return

• torch.Tensor

  • Encoded features (the codes).

sanitize_np_input(self,
                  x)

Ensure the input tensor is a numpy array of shape (batch_size, dims). Convert from pytorch tensor or DataLoader if necessary.

Parameters

• x : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, dims).

Return

• torch.Tensor

  • Sanitized input tensor.

fit(self,
    A,
    max_iter=500)

Fit the Semi-NMF model to the input data.

Parameters

• A : torch.Tensor or Iterable

  • Input tensor of shape (batch_size, n_features).

• max_iter : int, optional

  • Maximum number of iterations, by default 500.

init_random_d(self,
              A,
              Z)

Initialize the dictionary D using matrix inversion.

Parameters

• A : torch.Tensor

  • Input tensor of shape (batch_size, n_features).

• Z : torch.Tensor

  • Codes tensor of shape (batch_size, nb_concepts).

Return

• D : torch.Tensor

  • Initialized dictionary tensor.

sanitize_np_codes(self,
                  z)

Ensure the codes tensor (Z) is a numpy array of shape (batch_size, nb_concepts). Convert from pytorch tensor or DataLoader if necessary.

Parameters

• z : torch.Tensor

  • Encoded tensor (the codes) of shape (batch_size, nb_concepts).

Return

• torch.Tensor

  • Sanitized codes tensor.

decode(self,
       Z)

Decode the input tensor (the codes) using Semi-NMF.

Parameters

• Z : torch.Tensor

  • Encoded tensor (the codes) of shape (batch_size, nb_concepts).

Return

• torch.Tensor

  • Decoded output (the activations).

---

1. Convex and Semi-Nonnegative Matrix Factorizations by Ding, Li, and Jordan (2008). ↩