D-SLR Replaces Truncated SVD With Three Decompositions and No Tuning

Researchers have presented D-SLR (Disjoint Row-Sparse plus Low-Rank), a matrix compression method that aims to serve as a drop-in replacement for truncated SVD. It has a closed-form solution: the entire error-versus-parameter-count curve and the final solution cost three singular value decompositions, and with zero stored rows the algorithm reduces to ordinary SVD. That means that at an equal budget it does not lose to truncated SVD. The preprint appeared in arXiv cs.LG, and the experiments ran on LLM embedding tables, network traffic, and hyperspectral images.

What is wrong with truncated SVD

When a matrix has to be compressed with the ability to reconstruct it, truncated SVD is the default choice: the data is approximated with a single low-rank structure. The method is fast and well studied, but it has a weak spot. If a few rows fit poorly into the shared low-rank subspace, they drive up the residual error, and the rank has to be raised for all rows at once.

To fix this, a row-sparse component that overlaps with the low-rank part is usually added to the low-rank one. Such joint problems are solved iteratively, with regularization coefficients that need tuning. That means extra configuration, instability, and compute time. The authors of D-SLR propose to do without all of this.

How disjointness works

The key constraint is simple: each row is either stored verbatim or approximated by the low-rank fit, but never does both. For squared error, the authors say, this constraint costs nothing. The optimum of the joint problem is also reachable in the disjoint form, and with fewer parameters at any nontrivial rank and any number of stored rows.

The algorithm evaluates the whole tradeoff between error and parameter count at once, over a grid of rank × number of stored rows. The solution is chosen only afterward: by target error, by parameter budget, or by a given selection rule. A grid computed once can be reused for different requirements.

Key characteristics from the abstract:

• a closed-form solution with no iterative solvers;
• the cost of the grid and the solution together is three SVDs;
• no tunable parameters and no regularization;
• with zero stored rows, it coincides with truncated SVD;
• at every nontrivial rank and row count, it uses no more parameters than the joint model.

A certificate instead of hoping for a lucky choice

A separate contribution of the work is an a posteriori lower bound on the error that requires no assumptions about the data. It is computed for every solution shape, that is, for every pair of rank and number of stored rows. For a chosen solution, you can see how much any other combination could in principle gain. If the margin is small, there is no point in tuning further. If it is large, it is clear where to look.

The abstract gives no numerical results from the experiments. The authors write that experiments on synthetic and real data confirm the gain and show how informative the certificate is. The real scale of the improvement over SVD will have to be checked in the full text. The claim of "no worse at equal cost" follows from the construction of the method, but "how much better" depends on the structure of the specific data.

Where it can be applied

The most obvious candidate is the embedding tables of language models. In them, some tokens, for example rare or atypical ones, are poorly described by a shared low rank, and storing such rows in full may be cheaper than raising the rank for the whole table. Similar logic applies to network traffic with rare anomalous flows and to hyperspectral images with outlier pixels.

In practice, the method is worth testing wherever SVD already sits in a compression pipeline: it is enough to swap in D-SLR and compare the error curves at the same parameter budget. Because the result coincides with SVD at zero rows, the risk of regression is minimal. There is one caveat: the guarantees are formulated for squared error, so for model quality metrics, such as perplexity after weight compression, the gain has to be measured separately.