Cited By
View all- Li QLiang ZBernstein ADvorkin Y(2024)Revealing Decision Conservativeness Through Inverse Distributionally Robust OptimizationIEEE Control Systems Letters10.1109/LCSYS.2024.34076298(1018-1023)Online publication date: 2024
Data-Driven Inverse Optimization for Marginal Offer Price Recovery in Electricity Markets
Pages 497 - 509
Abstract
This paper presents a data-driven inverse optimization (IO) approach to recover the marginal offer prices of generators in a wholesale energy market. By leveraging underlying market-clearing processes, we establish a closed-form relationship between the unknown parameters and the publicly available market-clearing results. Based on this relationship, we formulate the data-driven IO problem as a computationally feasible single-level optimization problem. The solution of the data-driven model is based on the gradient descent method, which provides an error bound on the optimal solution and a sub-linear convergence rate. We also rigorously prove the existence and uniqueness of the global optimum to the proposed data-driven IO problem and analyze its robustness in two possible noisy settings. The effectiveness of the proposed method is demonstrated through simulations in both an illustrative IEEE 14-bus system and a realistic NYISO 1814-bus system.
References
[1]
Akshay Agrawal, Brandon Amos, Shane Barratt, Stephen Boyd, Steven Diamond, and J Zico Kolter. 2019. Differentiable convex optimization layers. In 33rd Conference on Neural Information Processing Systems (NeurIPS 2019). NIPS, Vancouver, Canada, 1–13.
[2]
Sara Ahmadian, Umang Bhaskar, Laura Sanità, and Chaitanya Swamy. 2018. Algorithms for inverse optimization problems. In 26th Annual European Symposium on Algorithms (ESA 2018). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, Helsinki, Finland, 1–14.
[3]
Ravindra K. Ahuja and James B. Orlin. 2001. Inverse optimization. Operations Research 49, 5 (2001), 771–783.
[4]
Brandon Amos and J. Zico Kolter. 2017. Optnet: Differentiable optimization as a layer in neural networks. In 34th International Conference on Machine Learning (ICML 2017). PMLR, Sydney, Australia, 136–145.
[5]
Anil Aswani, Zuo-Jun Shen, and Auyon Siddiq. 2018. Inverse optimization with noisy data. Operations Research 66, 3 (2018), 870–892.
[6]
Shahar Avin, Haydn Belfield, Miles Brundage, Gretchen Krueger, Jasmine Wang, Adrian Weller, Markus Anderljung, Igor Krawczuk, David Krueger, Jonathan Lebensold, 2021. Filling gaps in trustworthy development of AI. Science 374, 6573 (2021), 1327–1329.
[7]
Andreas Bärmann, Sebastian Pokutta, and Oskar Schneider. 2017. Emulating the expert: Inverse optimization through online learning. In 34th International Conference on Machine Learning (ICML 2017). PMLR, Sydney, Australia, 400–410.
[8]
Dimitris Bertsimas, Vishal Gupta, and Ioannis Ch Paschalidis. 2015. Data-driven estimation in equilibrium using inverse optimization. Mathematical Programming 153, 2 (2015), 595–633.
[9]
Yuexin Bian, Ningkun Zheng, Yang Zheng, Bolun Xu, and Yuanyuan Shi. 2022. Demand response model identification and behavior forecast with OptNet: a gradient-based approach. In 30th ACM International Conference on Future Energy Systems (ACM e-Energy 2022). ACM, virtual, USA, 418–429.
[10]
John R. Birge, Ali Hortaçsu, and J. Michael Pavlin. 2017. Inverse optimization for the recovery of market structure from market outcomes: An application to the MISO electricity market. Operations Research 65, 4 (2017), 837–855.
[11]
Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press, New York, USA.
[12]
Andrew Butler and Roy H Kwon. 2022. Efficient differentiable quadratic programming layers: an ADMM approach. Computational Optimization and Applications (2022), 1–28.
[13]
Timothy CY Chan and Neal Kaw. 2020. Inverse optimization for the recovery of constraint parameters. European Journal of Operational Research 282, 2 (2020), 415–427.
[14]
Timothy CY Chan, Taewoo Lee, and Daria Terekhov. 2019. Inverse optimization: Closed-form solutions, geometry, and goodness of fit. Management Science 65, 3 (2019), 1115–1135.
[15]
Ruidi Chen, Ioannis Ch. Paschalidis, and Michael C. Caramanis. 2017. Strategic equilibrium bidding for electricity suppliers in a day-ahead market using inverse optimization. In 56th IEEE Conference on Decision and Control (CDC 2017). IEEE, Melbourne, Australia, 220–225.
[16]
Ruidi Chen, Ioannis Ch. Paschalidis, Michael C. Caramanis, and Panagiotis Andrianesis. 2019. Learning from past bids to participate strategically in day-ahead electricity markets. IEEE Transactions on Smart Grid 10, 5 (2019), 5794–5806.
[17]
Chaosheng Dong, Yiran Chen, and Bo Zeng. 2018. Generalized inverse optimization through online learning. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018). NIPS, Montréal, Canada, 1–10.
[18]
B. Espen Eckbo. 2009. Bidding strategies and takeover premiums: A review. Journal of Corporate Finance 15, 1 (2009), 149–178.
[19]
Aaron Ferber, Bryan Wilder, Bistra Dilkina, and Milind Tambe. 2020. Mipaal: Mixed integer program as a layer. In 34th AAAI Conference on Artificial Intelligence (AAAI 2020). Springer, Thessaloniki, Greece, 1504–1511.
[20]
Illinois Center for a Smarter Electric Grid (ICSEG). 2022. IEEE 14-Bus System. University of Illinois at Urbana-Champaign. Retrieved Dec. 29, 2022 from https://icseg.iti.illinois.edu/ieee-14-bus-system/
[21]
Yong Fu and Mohammad Shahidehpour. 2007. Fast SCUC for large-scale power systems. IEEE Transactions on power systems 22, 4 (2007), 2144–2151.
[22]
Kimia Ghobadi, Taewoo Lee, Houra Mahmoudzadeh, and Daria Terekhov. 2018. Robust inverse optimization. Operations Research Letters 46, 3 (2018), 339–344.
[23]
Udi Helman. 2006. Market power monitoring and mitigation in the US wholesale power markets. Energy 31, 6-7 (2006), 877–904.
[24]
Elaheh H. Iraj and Daria Terekhov. 2021. Comparing inverse optimization and machine learning methods for imputing a convex objective function. arXiv preprint arXiv:2102.10742 (2021), 1–17.
[25]
Garud Iyengar and Wanmo Kang. 2005. Inverse conic programming with applications. Operations Research Letters 33, 3 (2005), 319–330.
[26]
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. 2021. Physics-informed machine learning. Nature Reviews Physics 3, 6 (2021), 422–440.
[27]
Arezou Keshavarz, Yang Wang, and Stephen Boyd. 2011. Imputing a convex objective function. In 2011 IEEE international symposium on intelligent control (ISIC 2011). IEEE, Denver, USA, 613–619.
[28]
András Kovács. 2021. Inverse optimization approach to the identification of electricity consumer models. Central European Journal of Operations Research 29, 2 (2021), 521–537.
[29]
Gong Li, Jing Shi, and Xiuli Qu. 2011. Modeling methods for GenCo bidding strategy optimization in the liberalized electricity spot market–A state-of-the-art review. Energy 36, 8 (2011), 4686–4700.
[30]
Houra Mahmoudzadeh and Kimia Ghobadi. 2022. Learning from Good and Bad Decisions: A Data-driven Inverse Optimization Approach. arXiv preprint arXiv:2207.02894 (2022), 1–36.
[31]
Robert Mieth, Yury Dvorkin, and Miguel A Ortega-Vazquez. 2022. Risk-Aware Dimensioning and Procurement of Contingency Reserve. IEEE Transactions on Power Systems (2022), 1–13.
[32]
Peyman Mohajerin Esfahani, Soroosh Shafieezadeh-Abadeh, Grani A Hanasusanto, and Daniel Kuhn. 2018. Data-driven inverse optimization with imperfect information. Mathematical Programming 167, 1 (2018), 191–234.
[33]
NYISO. 2022. NYISO market clearing results. NYISO. Retrieved Dec. 27, 2022 from http://mis.nyiso.com/public/P-27list.htm
[34]
NYISO. 2022. NYISO price data. NYISO. Retrieved Dec. 27, 2022 from https://www.nyiso.com/energy-market-operational-data
[35]
NYISO. 2022. Reliability Needs Assessment Report. NYISO. Retrieved Dec. 27, 2022 from https://www.nyiso.com/documents/20142/2248793/2022-RNA-Report.pdf
[36]
Richard P. O’Neill, Paul M. Sotkiewicz, Benjamin F. Hobbs, Michael H. Rothkopf, and William R. Stewart Jr. 2005. Efficient market-clearing prices in markets with nonconvexities. European journal of operational research 164, 1 (2005), 269–285.
[37]
NYISO Energy Market Operations. 2022. Day-Ahead Scheduling Manual. NYISO. Retrieved Jan. 2, 2023 from https://www.nyiso.com/documents/20142/2923301/dayahd_schd_mnl.pdf/0024bc71-4dd9-fa80-a816-f9f3e26ea53a
[38]
Remigijus Paulavičius and Julius Žilinskas. 2006. Analysis of different norms and corresponding Lipschitz constants for global optimization. Technological and Economic Development of Economy 12, 4 (2006), 301–306.
[39]
Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. 2011. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647 (2011), 1–21.
[40]
Simon Risanger, Stein-Erik Fleten, and Steven A. Gabriel. 2020. Inverse equilibrium analysis of oligopolistic electricity markets. IEEE Transactions on Power Systems 35, 6 (2020), 4159–4166.
[41]
Carlos Ruiz, Antonio J. Conejo, and Dimitris J. Bertsimas. 2013. Revealing rival marginal offer prices via inverse optimization. IEEE Transactions on Power Systems 28, 3 (2013), 3056–3064.
[42]
Javier Saez-Gallego and Juan M. Morales. 2017. Short-term forecasting of price-responsive loads using inverse optimization. IEEE Transactions on Smart Grid 9, 5 (2017), 4805–4814.
[43]
Javier Saez-Gallego, Juan M. Morales, Marco Zugno, and Henrik Madsen. 2016. A data-driven bidding model for a cluster of price-responsive consumers of electricity. IEEE Transactions on Power Systems 31, 6 (2016), 5001–5011.
[44]
NYISO Stakeholder Services. 2021. Guide 01 Market Participants User’s Guide. NYISO. Retrieved Jan. 2, 2023 from https://www.nyiso.com/documents/20142/3625950/mpug.pdf
[45]
Shai Shalev-Shwartz and Shai Ben-David. 2014. Understanding machine learning: From theory to algorithms. Cambridge university press, New York, USA.
[46]
Mathangi Srinivasan Kumar. 2022. Locational Based Marginal Pricing. NYISO. Retrieved Jan. 2, 2023 from https://www.nyiso.com/documents/20142/3037451/3-LMBP.pdf/f7682e03-e921-eaab-09bf-690524b5ade6
[47]
Yingcong Tan, Andrew Delong, and Daria Terekhov. 2019. Deep inverse optimization. In 16th International Conference on Integration of Constraint Programming, Artificial Intelligence, and Operations Research (CPAIOR 2019). Springer, Thessaloniki, Greece, 540–556.
[48]
Jérôme Thai and Alexandre M. Bayen. 2018. Imputing a variational inequality function or a convex objective function: A robust approach. J. Math. Anal. Appl. 457, 2 (2018), 1675–1695.
[49]
Trung Vu and Raviv Raich. 2022. On asymptotic linear convergence of projected gradient descent for constrained least squares. IEEE Transactions on Signal Processing 70 (2022), 4061–4076.
[50]
xxx. 2023. Data supplement: NYISO 1814-bus system. Retrieved Feb. 06, 2023 from https://github.com/xxx/data_supplement_NYISO
[51]
Jianzhong Zhang and Chengxian Xu. 2010. Inverse optimization for linearly constrained convex separable programming problems. European Journal of Operational Research 200, 3 (2010), 671–679.
Index Terms
- Data-Driven Inverse Optimization for Marginal Offer Price Recovery in Electricity Markets
Recommendations
Understanding Price Volatility in Electricity Markets
HICSS '00: Proceedings of the 33rd Hawaii International Conference on System Sciences-Volume 4 - Volume 4This paper illustrates notions of volatility associated with power systems spot prices for electricity. The paper demonstrates a frequency-domain method useful to separate out periodic price variations from random variations. It then uses actual ...
Portfolio Optimization in Incomplete Markets and Price Constraints Determined by Maximum Entropy in the Mean
AbstractA solution to a portfolio optimization problem is always conditioned by constraints on the initial capital and the price of the available market assets. If a risk neutral measure is known, then the price of each asset is the discounted expected ...
Blockchain and Federated Learning Based Bidding Applications in Power Markets
AbstractCurrently, power generators are facing an imperfectly competitive power market with relatively limited information, and they can increase their own economic returns by raising their quotations through game theory, which leads to an inflated price ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
June 2023
545 pages
ISBN:9798400700323
DOI:10.1145/3575813
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 16 June 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- US DOE ARPA-e
- NSF CMMI
Conference
e-Energy '23
Sponsor:
e-Energy '23: The 14th ACM International Conference on Future Energy Systems
June 20 - 23, 2023
FL, Orlando, USA
Acceptance Rates
Overall Acceptance Rate 160 of 446 submissions, 36%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 621Total Downloads
- Downloads (Last 12 months)339
- Downloads (Last 6 weeks)34
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
Cited By
View all- Li QLiang ZBernstein ADvorkin Y(2024)Revealing Decision Conservativeness Through Inverse Distributionally Robust OptimizationIEEE Control Systems Letters10.1109/LCSYS.2024.34076298(1018-1023)Online publication date: 2024
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatLogin options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in