Abstract
We consider robust empirical risk minimization (ERM), where model parameters are chosen to minimize the worst-case empirical loss when each data point varies over a given convex uncertainty set. In some simple cases, such problems can be expressed in an analytical form. In general the problem can be made tractable via dualization, which turns a min-max problem into a min-min problem. Dualization requires expertise and is tedious and error-prone. We demonstrate how CVXPY can be used to automate this dualization procedure in a user-friendly manner. Our framework allows practitioners to specify and solve robust ERM problems with a general class of convex losses, capturing many standard regression and classification problems. Users can easily specify any complex uncertainty set that is representable via disciplined convex programming (DCP) constraints.
This is a preview of subscription content, log in via an institution to check access.
Data availability
The data used to reproduce our results is available at https://zenodo.org/record/4446043#.Y9Y9ENJBwUE.
References
Aghasi, A., Feizollahi, M., Ghadimi, S.: Rigid: Robust linear regression with missing data (2022). https://arxiv.org/abs/2205.13635
Ben-Tal, A., El Ghaoui, L., Nemirovski, A.: Robust Optimization. Princeton University Press, Princeton (2009)
Bertsimas, D., Brown, D., Caramanis, C.: Theory and applications of robust optimization. SIAM Rev. 53(3), 464–501 (2011)
Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)
Diamond, S., Boyd, S.: CVXPY: a Python-embedded modeling language for convex optimization. J. Mach. Learn. Res. 17, 1–5 (2016)
El Ghaoui, L., Lebret, H.: Robust solutions to least-squares problems with uncertain data. SIAM J. Matrix Anal. Appl. 18, 1035–1064 (1997)
Gyódi, K., Nawaro, L.: Determinants of Airbnb prices in European cities: a spatial econometrics approach. Tour. Manag. 86, 104319 (2021)
Schiele, P., Luxenberg, E., Boyd, S.: Disciplined saddle programming. Transactions on Machine Learning Research, pp. 2835–8856 (2024). https://openreview.net/forum?id=KhMLfEIoUm
Shivaswamy, P.K., Bhattacharyya, C., Smola, A.J.: Second order cone programming approaches for 336 handling missing and uncertain data. J. Mach. Learn. Res. 7, 1283–1314 (2006)
Xu, H., Caramanis, C., Mannor, S.: Robust regression and lasso. In: Advances in Neural Information Processing Systems, pp. 1485–1510 (2008)
Xu, H., Caramanis, C., Mannor, S.: Robustness and regularization of support vector machines. J. Mach. Learn. Res. 10, 1485–1510 (2009)
Acknowledgements
Stephen Boyd was partially supported by ACCESS (AI Chip Center for Emerging Smart Systems), sponsored by InnoHK funding, Hong Kong SAR, and by Office of Naval Research Grant N00014-22-1-2121. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE1745016. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. This work was also supported by the AI Institute for Resilient Agriculture (NSF-USDA Award Number 2021-67021-35329).
Author information
Authors and Affiliations
Corresponding author
Additional information
Communicated by Olivier Fercoq.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
In this section, we present a helper function that automatically converts problems of the form (1) to problems of the form (3) and (4). This helper function requires the user to pass as inputs the CVXPY variables, constraints and loss function that define (1), and is presented below. The code is available at
We emphasize that the loss function f provided by the user is not verified to have the claimed curvature properties specified in the mode argument. It is impossible to verify this in general, so the user must be careful to provide a loss function that has the correct curvature properties. We also mention that the user is not limited to pass in a loss function f that is a CVXPY atom. Instead, the user has the flexibility to create a loss function that is a composition of several CVXPY atoms, as follows.
Rights and permissions
About this article
Cite this article
Luxenberg, E., Malik, D., Li, Y. et al. Specifying and Solving Robust Empirical Risk Minimization Problems Using CVXPY. J Optim Theory Appl 202, 1158–1168 (2024). https://doi.org/10.1007/s10957-024-02491-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10957-024-02491-6