capnet: Contribution-Capped Elastic Net Regularization

Harry He

2025-06-26

Method

The Contribution Cap Penalty

Consider a linear regression problem with elastic net regularization. The optimization objective is: \[\arg\min_{\beta}||Y-X\beta||_2^2+\lambda\Big(\alpha||\beta||_1+(1-\alpha)||\beta||_2^2\Big)\]

In many practical applications, it is desirable to impose additional structure on the model coefficients. One might want to enforce monotonicity, apply budget constraints, or—our focus here—limit the contribution of each covariate to the final prediction. This is especially relevant when:

We define the contribution of covariate \(j\) as the absolute value of the product \(\beta_j Z_j\), where \(Z\) is a matrix used to evaluate feature contributions (often the training data or a held-out evaluation set), and we penalize contributions that exceed a per-feature ceiling \(L_j\). This leads to a soft-constrained formulation: \[\min_{\beta_0,\beta}\frac{1}{2N}\sum_{i=1}^N\Big(y_i-(\beta_0+x_i^T\beta)\Big)^2+\lambda\bigg[\frac{(1-\alpha)}{2}||\beta||_2^2+\alpha||\beta||_1\bigg]+\gamma\sum_{j=1}^p\max\big(0,|\beta_jZ_j|-L_j\big)^2\]

where \(p\) is the number of covariates. When \(\gamma=0\) the model reduces to standard elastic net. The cap penalty uses squared violations (\(p=2\)), which is differentiable almost everywhere and facilitates convergence with quasi-Newton methods.

This formulation resembles generalized LASSO-type problems (Tibshirani and Taylor 2011) and constrained penalization frameworks (James, Paulson, and Rusmevichientong 2020), but the penalty is applied to feature contributions rather than coefficients themselves. Because the capped contributions involve data-dependent terms (\(\beta_j X_j\)), there is no simple matrix \(D\) such that the penalty reduces to a standard form \(\|D\beta\|_1\).

Optimization Strategy

The contribution cap penalty renders off-the-shelf solvers like glmnet unsuitable. Instead, capnet uses:

Both solvers are provided by the lbfgs package. The internal optimizer selection is automatic: orthantwise_c = lambda * alpha is passed to lbfgs(), which switches to OWL-QN whenever this is positive.


Getting Started

set.seed(123)
example <- simulate_capnet_data()
X <- example$X
y <- example$y
n <- nrow(X)
p <- ncol(X)
L <- example$L
guide_lambda <- exp(seq(1, -5, length.out = 25))
guide_alpha <- seq(0, 1, length.out = 5)
maxx <- t(as.matrix(apply(X, 2, function(x) {
  mx <- max(x)
  mn <- min(x)
  if (abs(mx) > abs(mn)) mx else mn
})))
beta_init <- rep(0, p + 1)

The simulator returns a 100$$20 input matrix X, response vector y, and a contribution cap vector L. maxx collects the maximum-magnitude value in each column—used later to evaluate caps against the worst-case observation.

Cross-Validation and Fitting

Use cv_capnet to search over \(\alpha\) and \(\lambda\) via K-fold cross-validation, then pass the selected hyperparameters to capnet:

set.seed(1)
capnet.cv1 <- cv_capnet(
  X, y, 0, L, lambda = guide_lambda, alpha = guide_alpha,
  standardize = FALSE
)
alpha1 <- capnet.cv1$best_alpha
lambda1 <- capnet.cv1$best_lambda
capnet.mod1 <- capnet(X, y,
  alpha = alpha1, lambda = lambda1,
  gamma = 0, L = L, standardize = FALSE
)

Best alpha value: 1.000 | Best lambda value 0.0498

Adding a contribution cap penalty (\(\gamma > 0\)) may shift the selected \(\lambda\):

set.seed(2)
capnet.cv2 <- cv_capnet(
  X, y, 1, L, lambda = guide_lambda, alpha = guide_alpha,
  standardize = FALSE
)
alpha2 <- capnet.cv2$best_alpha
lambda2 <- capnet.cv2$best_lambda
capnet.mod2 <- capnet(X, y,
  alpha = alpha2, lambda = lambda2,
  gamma = 0.5, L = L, standardize = FALSE
)

Best alpha value: 0.250 | Best lambda value 0.0067

Visualizing Coefficient Redistribution

plot_redistribution compares two fitted models side by side, showing both raw coefficient values and the implied per-feature contributions:

plots <- plot_redistribution(capnet.mod1, capnet.mod2, plot = "raw")
plots$beta

plots$contribution

Computational Cost

capnet is slower than glmnet because it relies on a general-purpose quasi-Newton optimizer rather than the coordinate descent path algorithm. The following timings give a rough sense of overhead on the simulated data:

## [1] "System runtime for unconstrained 5-fold cross-validation"
##    user  system elapsed 
##   6.773   0.309   7.089
## [1] "System runtime for constrained 5-fold cross-validation"
##    user  system elapsed 
##   6.839   0.327   7.175
## [1] "System runtime for constrained model fit"
##    user  system elapsed 
##   0.023   0.000   0.024

Model Parameters

Standardization

Like glmnet, capnet accepts a standardize argument (default TRUE) that scales the columns of X before fitting. The cap penalty is always evaluated at the original (unstandardized) scale: internally, coefficients are converted back to raw scale before computing cap violations, and the resulting gradient is chain-rule-corrected back into standardized space. This means z (the evaluation matrix) and L should always be supplied in the same scale as X, regardless of standardize.

Multiplier

The multiplier argument scales feature contributions row-wise during cap enforcement only; it does not affect the likelihood. When contributions must be evaluated at multiple prediction points with different per-observation scaling factors, pass multiplier as an \(n \times 1\) vector. This avoids the need to reshape L into a matrix aligned with each observation.

Box Constraints

upper.limits and lower.limits enforce hard bounds on individual coefficients via gradient masking: if a parameter sits within tol of its bound and the gradient would push it further out of bounds, that gradient entry is zeroed. Increase tol if the default fails to enforce the constraint; decrease it if updates are suppressed prematurely. Because masking is applied to gradients (not the iterates themselves), initial parameter values must lie within the specified bounds—the default zero initialization satisfies symmetric bounds.

Diagnosing Convergence Failure

capnet does not stop or raise an error when the optimizer fails to converge. Inspect fit$convergence and fit$message after fitting: a value other than 0 (or the acceptable rounding-error code -1001) indicates the optimizer stopped early. A convergence code of \(-998\) typically means the line search failed because the gradient became too large or numerically unstable. Reducing gamma, reducing lambda, or rescaling ill-conditioned features can help.


Convergence Sensitivity to \(\gamma\)

Convergence Across Specifications

The contribution cap optimizer introduces a new hyperparameter \(\gamma\) in addition to the \(\alpha\) and \(\lambda\) parameters of the elastic net fitter. A large or ill-conceived \(\gamma\) value can force the line search algorithm in the underlying OWL-QN method to fail. To prevent scaling mismatch, capnet normalizes the cap loss and gradient so that they are in the same units as the rest of the objective. Nevertheless, the algorithm can fail at large \(\gamma\), and the failure threshold depends heavily on the data and specification. The following table shows convergence results for eight specifications across a wide range of \(\gamma\) values.

for (i in seq(-5, 10)) {
  # Regular model
  model1 <- capnet(X, y,
    alpha = 0.5, lambda = 0.01,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
  )
  convergence1 <- model1$convergence

  # Larger lambda model
  model2 <- capnet(X, y,
    alpha = 0.5, lambda = 0.5,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
  )
  convergence2 <- model2$convergence

  # No elastic net penalty model
  model3 <- capnet(X, y,
    alpha = 0.5, lambda = 0,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
  )
  convergence3 <- model3$convergence

  # Ridge regression model
  model4 <- capnet(X, y,
    alpha = 0, lambda = 0.01,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
  )
  convergence4 <- model4$convergence

  # LASSO regression model
  model5 <- capnet(X, y,
    alpha = 1, lambda = 0.01,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 1
  )
  convergence5 <- model5$convergence

  # Large multiplier model
  model6 <- capnet(X, y,
    alpha = 0.5, lambda = 0.01,
    gamma = 10**i, L = L, z = X, standardize = TRUE, multiplier = 100
  )
  convergence6 <- model6$convergence

  # Small L model
  model7 <- capnet(X, y,
    alpha = 0.5, lambda = 0.01,
    gamma = 10**i, L = L / 2, z = X, standardize = TRUE, multiplier = 1
  )
  convergence7 <- model7$convergence

  # Single max-row model
  model8 <- capnet(X, y,
    alpha = 0.5, lambda = 0.01,
    gamma = 10**i, L = L, z = maxx, standardize = TRUE, multiplier = 1
  )
  convergence8 <- model8$convergence

  convergence.table <- rbind(
    convergence.table,
    c(
      convergence1, convergence2,
      convergence3, convergence4,
      convergence5, convergence6,
      convergence7, convergence8
    )
  )
}
Gamma Base Large Lambda No Elastic Net Ridge LASSO Large Multiplier Small L Single Max Row
1e-05 0 0 0 0 0 0 0 0
1e-04 0 0 0 0 0 0 0 0
1e-03 0 0 0 0 0 0 0 0
1e-02 0 0 0 0 0 0 0 0
1e-01 0 0 0 0 0 0 0 0
1e+00 0 0 0 0 0 0 0 0
1e+01 0 0 0 0 0 0 0 0
1e+02 0 0 0 0 0 0 0 0
1e+03 0 0 -1001 0 0 0 0 0
1e+04 0 0 -1001 -1001 0 0 0 0
1e+05 0 0 -1001 -1001 0 0 0 -998
1e+06 0 0 -1001 -1001 0 0 0 -998
1e+07 -998 0 -1001 -1001 -998 0 0 -998
1e+08 -998 -998 -1001 -1001 -998 -998 -998 -998
1e+09 -998 -998 -1001 -1001 -998 -998 -998 -998
1e+10 -998 -998 -1001 -1001 -998 -998 -998 -998

Why Different Specifications Fail at Different \(\gamma\) Values

Most configurations tolerate a wide range of \(\gamma\) values before failing. The failure threshold varies systematically across specifications for reasons rooted in the optimizer’s sensitivity to sudden large gradient changes. The cap penalty gradient has the form

\[\frac{2\gamma}{m} \sum_{i} e_{ij} \cdot \operatorname{sgn}(x_{ij}\hat\beta_j) \cdot \frac{x_{ij}}{\sigma_j}\]

where \(e_{ij} = \max(0,\,|x_{ij}\hat\beta_j| - L_j)\) is the per-observation cap excess and \(m = \text{nrow}(\mathbf{z})\). When this gradient becomes very large relative to the likelihood gradient, L-BFGS and OWL-QN accumulate a corrupted inverse-Hessian approximation over successive iterations, causing the line search to fail.

Optimizer type (Ridge and No Elastic Net fail earliest). The underlying optimizer is selected via orthantwise_c \(= \lambda\alpha\) passed to lbfgs(). When \(\alpha = 0\) (Ridge) or \(\lambda = 0\) (No Elastic Net), orthantwise_c \(= 0\) and the solver falls back to standard L-BFGS. All other configurations with \(\alpha > 0\) and \(\lambda > 0\) use OWL-QN, which enforces an orthant constraint at each step: coefficients cannot cross zero within a single iteration. This constraint bounds step sizes and prevents the large overshoots that corrupt the curvature history when \(\gamma\) is large. L-BFGS has no such bound and therefore degrades faster, explaining why Ridge and No Elastic Net fail at lower \(\gamma\) than the OWL-QN models.

Number of evaluation rows \(m\) (Single Max Row fails earliest overall). The cap gradient is divided by \(m\), the number of rows in \(\mathbf{z}\). With \(m = n\) (models 1–7, \(\mathbf{z} = X\)), this average moderates the gradient magnitude: each observation contributes only \(1/n\) of its excess to the gradient. With \(m = 1\) (model 8, \(\mathbf{z} = \mathbf{x}_{\max}\)), the full excess of the single worst-case row enters the gradient undiluted, making the cap gradient \(n\) times larger for the same \(\gamma\). This is why model 8 fails at a \(\gamma\) roughly \(n\) times smaller than the baseline.

Effective cap threshold (Large Multiplier and Small L fail earlier). A multiplier \(> 1\) tightens the effective cap: the excess term becomes \(\max(0,\,|\text{multiplier}_i \cdot x_{ij}\hat\beta_j| - L_j) / \text{multiplier}_i = \max(0,\,|x_{ij}\hat\beta_j| - L_j/\text{multiplier}_i)\), so multiplier = 100 reduces the effective ceiling from \(L_j\) to \(L_j/100\). Similarly, halving \(L\) (Small L) directly tightens the cap. Both changes produce more cap violations and larger excess values during optimization, increasing the cap gradient magnitude and lowering the failure threshold.

Regularization strength (Larger \(\lambda\) tolerates higher \(\gamma\)). Stronger elastic net regularization keeps \(\hat\beta\) smaller throughout the optimization path, which reduces cap violations and therefore the magnitude of the cap gradient. A model with \(\lambda = 0.5\) reaches the cap boundary later (at higher \(\gamma\)) than the baseline \(\lambda = 0.01\), while \(\lambda = 0\) (No Elastic Net) eliminates all coefficient-shrinkage pressure and increases violations, shifting the failure threshold downward.


\(\lambda\) Selection with Active Contribution Caps

Cross-Validation Instability

The elastic net regularization and contribution cap constraints are correlated with each other despite serving different roles: both shrink coefficients and reduce realized contributions. This interaction can destabilize cross-validation. The following experiment runs 10 independent CV rounds at three \(\gamma\) values to illustrate the effect. Each run uses five folds and six \(\lambda\) values, with the maximum-magnitude row of \(X\) as the evaluation matrix so cap violations are activated across folds.

K <- 5L
n_runs <- 10L
lambda_grid <- 10^seq(-11, -1, length.out = 6)

run_one <- function(i) {
  set.seed(i)
  cv0 <- cv_capnet(X, y,
    gamma = 0, L = L,
    alpha = 0.5, lambda = lambda_grid,
    z = maxx, K = K, standardize = TRUE, multiplier = 1
  )
  cv01 <- cv_capnet(X, y,
    gamma = 0.1, L = L,
    alpha = 0.5, lambda = lambda_grid,
    z = maxx, K = K, standardize = TRUE, multiplier = 1
  )
  cv100 <- cv_capnet(X, y,
    gamma = 100, L = L,
    alpha = 0.5, lambda = lambda_grid,
    z = maxx, K = K, standardize = TRUE, multiplier = 1
  )

  fit0 <- capnet(X, y,
    alpha = 0.5, lambda = cv0$best_lambda,
    gamma = 0, L = L, z = maxx, standardize = TRUE, multiplier = 1
  )
  fit01 <- capnet(X, y,
    alpha = 0.5, lambda = cv01$best_lambda,
    gamma = 0.1, L = L, z = maxx, standardize = TRUE, multiplier = 1
  )
  fit100 <- capnet(X, y,
    alpha = 0.5, lambda = cv100$best_lambda,
    gamma = 100, L = L, z = maxx, standardize = TRUE, multiplier = 1
  )

  list(
    lambda0 = cv0$best_lambda, beta0 = as.numeric(fit0$beta),
    lambda01 = cv01$best_lambda, beta01 = as.numeric(fit01$beta),
    lambda100 = cv100$best_lambda, beta100 = as.numeric(fit100$beta)
  )
}

results <- lapply(seq_len(n_runs), run_one)

lambdas0 <- sapply(results, `[[`, "lambda0")
lambdas01 <- sapply(results, `[[`, "lambda01")
lambdas100 <- sapply(results, `[[`, "lambda100")

betas0 <- do.call(rbind, lapply(results, `[[`, "beta0"))
betas01 <- do.call(rbind, lapply(results, `[[`, "beta01"))
betas100 <- do.call(rbind, lapply(results, `[[`, "beta100"))

When \(\gamma = 0\), cross-validation reliably identifies a concentrated range of optimal \(\lambda\) values. When a small \(\gamma = 0.1\) is introduced, the selected \(\lambda\) values spread and generally shift lower—the model learns to rely on contribution caps rather than \(\ell_1\)/\(\ell_2\) norms for shrinkage. At \(\gamma = 100\), the distribution becomes bimodal: some runs pick a very small \(\lambda\) (caps dominate regularization) while others pick a very large one (elastic net shrinks coefficients so aggressively that cap violations rarely occur). This bimodality arises because the two penalties are redundant—both shrink coefficients and reduce excess contributions—so the CV error surface flattens along the \(\lambda\) axis. Under random fold splits, the argmin can jump between the two grid extremes. Because this instability stems from correlated regularization, running cross-validation on \(\lambda\) while holding \(\gamma\) fixed can produce unreliable results; users should fix \(\gamma\) based on domain knowledge and treat it as a model-design choice rather than a tuning parameter.

Coefficient Stability Across CV Runs

Although different CV rounds may select very different \(\lambda\) values when caps are active, the downstream coefficient estimates remain stable. The following plot shows the mean coefficient (solid line) and the full range (min-to-max band) across the 50 runs for each \(\gamma\) level.

The narrow confidence bands—barely visible even at \(\gamma = 100\)—confirm that while cross-validation selects very different \(\lambda\) values across runs, the model coefficients are nearly identical. This is consistent with the correlated-regularization interpretation: the elastic net and cap penalties trade off against each other along a near-flat objective surface, so the optimizer reaches essentially the same point regardless of which end of the \(\lambda\) grid is chosen.


Predictive Performance

Standard Well-Behaved Features

The first experiment uses the QuickStartExample data from glmnet. All features are well-behaved with stable variance, so cap violations should be rare and the three models should perform similarly.

ds_qs <- prepare_data("quickstart")
set.seed(123)
fits_qs <- fit_models(
  ds_qs$train_X, ds_qs$train_y, ds_qs$test_X, ds_qs$L,
  ds_qs$gamma
)

qs_enet_is <- mse_from_beta(
  ds_qs$train_X, ds_qs$train_y, fits_qs$enet$beta0,
  fits_qs$enet$beta
)
qs_cap1_is <- mse_from_beta(
  ds_qs$train_X, ds_qs$train_y, fits_qs$cap1$beta0,
  fits_qs$cap1$beta
)

qs_enet_oos <- mse_from_beta(
  ds_qs$test_X, ds_qs$test_y, fits_qs$enet$beta0,
  fits_qs$enet$beta
)
qs_cap1_oos <- mse_from_beta(
  ds_qs$test_X, ds_qs$test_y, fits_qs$cap1$beta0,
  fits_qs$cap1$beta
)
qs_cap2_oos <- mse_from_pred(fits_qs$cap2$predictions, ds_qs$test_y)

report_qs <- data.frame(
  Model = c("Elastic Net (post-fit cap)", "Optimized Cap", "Walk Cap"),
  IS = c(qs_enet_is, qs_cap1_is, NA),
  OOS = c(qs_enet_oos, qs_cap1_oos, qs_cap2_oos)
)
knitr::kable(report_qs, digits = 6)
Model IS OOS
Elastic Net (post-fit cap) 0.501179 0.591483
Optimized Cap 0.495869 0.580050
Walk Cap NA 0.538647

The in-sample (IS) and out-of-sample (OOS) MSEs show that the elastic net and optimized cap models perform comparably. The walk-forward model achieves slightly better OOS performance, attributable to its per-observation refitting—but at higher computational cost and without a single stable coefficient vector.

High-Variance Features with Occasional Spikes

To illustrate the strength of capnet, the second experiment uses a synthetic latent data-generating process \(Z \rightarrow Y\), where the true predictors \(Z\) drive the outcome but only a noisy proxy \(X\) is observed. Some columns of \(X\) have unstable variance—occasionally taking extreme values with probability high_prob. Under these conditions, a few extreme observations can dominate an elastic net fit, inflating OOS error. The contribution cap explicitly limits how much any single feature can contribute, regardless of the magnitude of its realized value.

ds_px <- prepare_data("proxy")
set.seed(123)
fits_px <- fit_models(
  ds_px$train_X, ds_px$train_y, ds_px$test_X, ds_px$L,
  ds_px$gamma
)

px_enet_is <- mse_from_beta(
  ds_px$train_X, ds_px$train_y, fits_px$enet$beta0,
  fits_px$enet$beta
)
px_cap1_is <- mse_from_beta(
  ds_px$train_X, ds_px$train_y, fits_px$cap1$beta0,
  fits_px$cap1$beta
)

px_enet_oos <- mse_from_beta(
  ds_px$test_X, ds_px$test_y, fits_px$enet$beta0,
  fits_px$enet$beta
)
px_cap1_oos <- mse_from_beta(
  ds_px$test_X, ds_px$test_y, fits_px$cap1$beta0,
  fits_px$cap1$beta
)
px_cap2_oos <- mse_from_pred(fits_px$cap2$predictions, ds_px$test_y)

report_px <- data.frame(
  Model = c("Elastic Net (post-fit cap)", "Optimized Cap", "Walk Cap"),
  IS = c(px_enet_is, px_cap1_is, NA),
  OOS = c(px_enet_oos, px_cap1_oos, px_cap2_oos)
)
knitr::kable(report_px, digits = 6)
Model IS OOS
Elastic Net (post-fit cap) 0.860357 0.795325
Optimized Cap 0.803383 0.745417
Walk Cap NA 0.686337

Both cap models outperform the elastic net baseline on OOS MSE. The walk-forward implementation achieves the greatest improvement because it refits on the most recent window before each prediction, adapting to any shift in the spike pattern seen out of sample.


Application Domains

Finance

Social Science

Machine Learning and Statistical Modeling

Healthcare and Natural Science

Andrew, Galen, and Jianfeng Gao. 2007. “Scalable Training of L1-Regularized Log-Linear Models.” In Proceedings of the 24th International Conference on Machine Learning, 33–40. ICML ’07. New York, NY, USA.
James, Gareth M., Courtney Paulson, and Paat Rusmevichientong. 2020. “Penalized and Constrained Optimization: An Application to High-Dimensional Website Advertising.” Journal of American Statistical Association 115 (529): 107–22.
Tibshirani, Ryan J., and Jonathan Taylor. 2011. “The Solution Path of the Generalized LASSO.” The Annals of Statistics 39 (3): 1335–71.