The contribution cap optimizer introduces a new hyperparameter \(\mu\) in addition to the \(\alpha\) and \(\lambda\) parameters of the elastic net
fitter. Because the new added contribution cap penalty term is smooth
almost everywhere except at the border, a large or ill-conceived \(\mu\) value can for the line search
algorithm in the underlying Orthant-Wise Limited-memory Quasi Newtonian
(OWL-QN) method to fail, resulting in convergence issue. To prevent
scaling mismatch between the contribution cap penalty term (which is
designed to be in the original scale even when standardize
is set to TRUE), I scale the contribution cap loss and
gradient in the same unit as the rest of the loss and gradient functions
in the capnet() function. Nevertheless, the algorithm can
fail under some \(\mu\) values, and the
users should pay careful attention as the algorithm will not report a
warning or error by default. The following code illustrates an example
of how the optimizer reacts to different values of \(\mu\). Note that the available range of
\(\mu\) is sensitive to the input
data
example <- simulate_capnet_data()
X <- example$X
y <- example$y
n <- nrow(X)
p <- ncol(X)
L <- example$L
maxx <- t(as.matrix(apply(X, 2, function(x) {
mx <- max(x)
mn <- min(x)
if (abs(mx) > abs(mn)) mx else mn
})))
beta_init <- rep(0, p + 1)The following table shows convergence results with eight different sets of parameter setups. The first model is a regular baseline model. The second model uses a larger \(\lambda\) value for greater elastic net penalty. The third model sets \(\lambda\) to \(0\), removing the L1 and L2 regularization terms. Model 4 and 5 are ridge and LASSO models. Model 6 incorporates a larger multiplier term. Model 7 chooses a smaller contribution cap, forcing the parameters to be even smaller. The last model uses a different evaluation matrix, which takes the maximum absolute value. This forces the model to be essentially under the same constraint but shrinks the number of observation to one.
for (i in seq(-5, 10)) {
# Regular model
model1 <- capnet(X, y, alpha = 0.5, lambda = 0.01,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 1)
convergence1 <- model1$convergence
# Larger lambda model
model2 <- capnet(X, y, alpha = 0.5, lambda = 0.5,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 1)
convergence2 <- model2$convergence
# No elastic net penalty model
model3 <- capnet(X, y, alpha = 0.5, lambda = 0,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 1)
convergence3 <- model3$convergence
# Ridge regression model
model4 <- capnet(X, y, alpha = 0, lambda = 0.01,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 1)
convergence4 <- model4$convergence
# LASSO regression model
model5 <- capnet(X, y, alpha = 1, lambda = 0.01,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 1)
convergence5 <- model5$convergence
# Large multiplier model
model6 <- capnet(X, y, alpha = 0.5, lambda = 0.01,
mu = 10 ** i, L = L, newx = X, standardize = TRUE, multiplier = 100)
convergence6 <- model6$convergence
# Small L model
model7 <- capnet(X, y, alpha = 0.5, lambda = 0.01,
mu = 10 ** i, L = L/2, newx = X, standardize = TRUE, multiplier = 1)
convergence7 <- model7$convergence
# Different `newx` model
model8 <- capnet(X, y, alpha = 0.5, lambda = 0.01,
mu = 10 ** i, L = L, newx = maxx, standardize = TRUE, multiplier = 1)
convergence8 <- model8$convergence
convergence.table <- rbind(convergence.table,
c(convergence1, convergence2,
convergence3, convergence4,
convergence5, convergence6,
convergence7, convergence8))
}| Mu | Base | Large Lambda | No Elastic Net | Ridge | LASSO | Large Multiplier | Small L | Single Max Row |
|---|---|---|---|---|---|---|---|---|
| 1e-05 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-04 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-02 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e-01 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+00 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+01 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+02 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+03 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+04 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 1e+05 | 0 | 0 | -1001 | -1001 | 0 | 0 | 0 | -998 |
| 1e+06 | 0 | 0 | -1001 | -1001 | 0 | 0 | -998 | -998 |
| 1e+07 | -998 | 0 | -1001 | -1001 | 0 | 0 | -998 | -998 |
| 1e+08 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
| 1e+09 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
| 1e+10 | -998 | -998 | -1001 | -1001 | -998 | -998 | -998 | -998 |
The table show that the model encounters no convergence failure with most \(\mu\) values, but when \(\mu\) becomes extremely large (greater than \(10^5\) or \(10^6\)), convergence may fail. It is also worth noting that when the L1 term is dropped in the loss function, the model has trouble converging with smaller values of \(\mu\). Changing \(\lambda\) and multiplier values do not significantly impact model’s ability to converge. Finally, if we force the same contribution cap constraints but aggregate evaluation matrix to one row of observation, the model is more likely to encounter convergence failure with larger \(\mu\) values. This is because the optimizer takes an average of all contribution cap excess when calculating the loss and gradient values, so having more observations moderates the strength of contribution cap panelty in each iteration, leading to easier and smoother convergence.
lambdas0 <- lambdas10 <- lambdas1000 <- c()
for (i in 1:50) {
lambda0.tmp <- cv_capnet(X, y, mu = 0, L = L,
alpha = 0.5, lambda = 10^seq(-7, -1, length.out = 7),
newx = maxx, standardize = TRUE, multiplier = 1)$best_lambda
lambda10.tmp <- cv_capnet(X, y, mu = 10, L = L,
alpha = 0.5, lambda = 10^seq(-7, -1, length.out = 7),
newx = maxx, standardize = TRUE, multiplier = 1)$best_lambda
lambda1000.tmp <- cv_capnet(X, y, mu = 1000, L = L,
alpha = 0.5, lambda = 10^seq(-7, -1, length.out = 7),
newx = maxx, standardize = TRUE, multiplier = 1)$best_lambda
lambdas0 <- c(lambdas0, lambda0.tmp)
lambdas10 <- c(lambdas10, lambda10.tmp)
lambdas1000 <- c(lambdas1000, lambda1000.tmp)
}Clearly, we see divergence in the optimal \(\mu\) value from cross-validation when the contribution cap penalty is activated. This can potentially be explained by the fact that the contribution penalty also restrains the size of the parameters, and by imposing a constraint on the size of the feature contributions, the user is also punishing parameters from being too large. Setting \(\lambda\) too large when contribution cap is present may over-penalize the model and lead to worse performances.
##
## Welch Two Sample t-test
##
## data: lambdas0 and lambdas10
## t = -2.951, df = 49.367, p-value = 0.004835
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.030625342 -0.005814758
## sample estimates:
## mean of x mean of y
## 0.009222004 0.027442054
##
## Welch Two Sample t-test
##
## data: lambdas10 and lambdas1000
## t = -2.989, df = 96.626, p-value = 0.003549
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## -0.046190483 -0.009325477
## sample estimates:
## mean of x mean of y
## 0.02744205 0.05520003
The t-tests also confirm the observation; however, we do not observe meaningful differences in \(\lambda\) values between models with \(\mu\) values set to \(10\) and \(1000\) respectively. This means that the optimal \(\mu\) value is not as sensitive to the strength of the contribution cap penalty.
capmod1 <- capnet(X, y, alpha = 0.5, lambda = mean(lambdas0),
mu = 100, L = L, newx = maxx)
capmod2 <- capnet(X, y, alpha = 0.5, lambda = mean(lambdas1000),
mu = 100, L = L, newx = maxx)
print(capmod1$value / capmod2$value)## [1] 0.9371463
The loss is apparently when choosing the less optimal \(\lambda\) value. The practical implication
fo this is that the user should be cautious using \(\lambda\) values derived from functions
such as glmnet (which conforms with the \(\mu=0\) case) when fitting a contribution
cap model.