Abstract
Machine learning and particularly reinforcement learning methods may be applied to control tasks ranging from single control loops to the operation of whole production plants. However, their utilization in industrial contexts lacks understandability and requires suitable levels of operability and maintainability. In order to asses different application scenarios a simple measure for their complexity is proposed and evaluated on four examples in a simulated palette transport system of a cold rolling mill. The measure is based on the size of controller input and output space determined by different granularity levels in a hierarchical process control model. The impact of these decomposition strategies on system characteristics, especially operability and maintainability, are discussed, assuming solvability and a suitable quality of the reinforcement learning solution is provided.
Zusammenfassung
Methoden des maschinellen Lernens, insbesondere des bestärkenden Lernens, könnten auf Automatisierungsaufgaben vom einzelnen Regelkreis bis hin zum Betrieb ganzer Anlagen angewendet werden. Im industriellen Kontext müssen sie dabei verständlich sein und ein geeignetes Maß an Betreib- und Wartbarkeit unterstützen. Um verschiedene Einsatzgebiete zu beurteilen wird daher ein einfaches Maß für die Komplexität vorgestellt und an vier Beispielen für ein simuliertes Palettentransportsystem eines Kaltwalzwerks dargestellt. Das Komplexitätsmaß basiert auf der Größe der Wertebereiche der Ein- und Ausgänge der Führungsfunktion, die durch verschiedene Granularitätsstufen eines hierarchischen Prozessführungsmodells festgelegt werden. Unter der Annahme der Lösbarkeit der Prozessführungsaufgabe mittels bestärkenden, maschinellen Lernens in einer geeigneten Güte wird der Einfluss dieser Dekompositionsstrategien auf Systemeigenschaften, insbesondere die Betreib- und Wartbarkeit aufgezeigt.
Funding source: Bundesministerium für Bildung und Forschung
Award Identifier / Grant number: 01IS19022
Funding statement: The research leading to these results has been funded by the German Federal Ministry of Education and Research (BMBF) under grant agreement no. 01IS19022 (BaSys 4.2).
About the authors
Julian Grothoff, M. Sc. RWTH (born 1990) is a research associate at the Chair of Information and Automation Systems for Process and Material Technology at RWTH Aachen University since 2016. His research focuses on the component-oriented and model-based realization of process control architectures with an emphasis on the integration of AI methods.
Nicolas Camargo Torres, M. Sc. RWTH (born 1994) has been a research associate at the Chair of Information and Automation Systems for Process and Material Technology at RWTH Aachen University in 2021. His research focuses on the application of AI methods for the automatic generation of automation applications with a focus on the understandability of the AI solution.
Prof. Dr.-Ing. Tobias Kleinert (born 1971) graduated in Mechanical Engineering in 1999 at RWTH Aachen University and completed his PhD in 2005 at the Chair of Automation and Computer Control of Prof. Jan Lunze at Ruhr-Universität Bochum. His career led him to BASF SE where he worked in the areas of Advanced Process Control, Production Technology Propylene Oxide, Regulated Automation Solutions, Digital Control Systems, Manufacturing Execution Solutions and Smart Manufacturing. As senior manager automation and digitalization he had assignments at the BASF sites in Ludwigshafen/D, Antwerp/B and Schwarzheide/D. Since 2020, he leads the Chair of Information and Automation Systems for Process and Material Technology at RWTH Aachen University with a focus on information processing, automation and digitalization.
References
1. Barredo-Arrieta, A., I. Laña and J. Del Ser. 2019. What lies beneath: A note on the explainability of black-box machine learning models for road traffic forecasting. In: Intelligent Transportation Systems Conference (ITSC).10.1109/ITSC.2019.8916985Search in Google Scholar
2. Dann, C. and E. Brunskill. 2015. Sample Complexity of Episodic Fixed-Horizon Reinforcement Learning. arXiv preprint.Search in Google Scholar
3. Elfaham, H. and U. Epple. 2020. Meta Models for Intralogistics. at – Automatisierungstechnik 68(3): 208–221.10.1515/auto-2019-0083Search in Google Scholar
4. Furuta, H., T. Matsushima, T. Kozuno, Y. Matsuo, S. Levine, O. Nachum and S. S. Gu. 2021. Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learning. arXiv preprint.Search in Google Scholar
5. Gazzaneo, V., J. C. Carrasco, D. R. Vinson and F. V. Lima. 2019. Process Operability Algorithms: Past, Present, and Future Developments. Industrial & Engineering Chemistry Research 59(6): 2457–2470.10.1021/acs.iecr.9b05181Search in Google Scholar
6. Grothoff, J. and H. Elfahaam. 2020. Interoperabilität und Wandelbarkeit in Cyber-Physischen-Produktionssystemen durch modulare Prozessführungs-Komponenten. In: Handbuch Industrie 4.0, Springer Reference Technik.10.1007/978-3-662-45537-1_144-1Search in Google Scholar
7. Grothoff, J. and T. Kleinert. 2020. Mapping of Standardized State Machines to Utilize Machine Learning Models in Process Control Environments. In: Cybersecurity workshop by European Steel Technology Platform.10.1007/978-3-030-69367-1_4Search in Google Scholar
8. Grothoff, J., C. Wagner and U. Epple. 2018. BaSys 4.0: Metamodell der Komponenten und Ihres Aufbaus. Publikationsserver der RWTH Aachen University, Aachen.Search in Google Scholar
9. Guidotti, R., A. Monreale, S. Ruggieri, F. Turini, F. Giannotti and D. Pedreschi. 2019. A Survey of Methods for Explaining Black Box Models. ACM Comput. Surv. 51(5): 1–41.10.1145/3236009Search in Google Scholar
10. Heuillet, A., F. Couthouis and N. Díaz-Rodríguez. 2021. Explainability in deep reinforcement learning. Knowledge-Based Systems 214: 106685.10.1016/j.knosys.2020.106685Search in Google Scholar
11. Islam, S. R., W. Eberle and S. K. Ghafoor. 2020. Towards quantification of explainability in explainable artificial intelligence methods. In The Thirty-Third International Flairs Conference.Search in Google Scholar
12. Kearns, M. and S. Singh. 2002. Near-Optimal Reinforcement Learning in Polynomial Time. Machine Learning 492: 209–232.10.1023/A:1017984413808Search in Google Scholar
13. Koenig, S. and R. G. Simmons. 1993. Complexity Analysis of Real-Time Reinforcement Learning. In: AAAI, pp. 99–107.Search in Google Scholar
14. Lattimore, T., M. Hutter and P. Sunehag. 2013. The Sample-Complexity of General Reinforcement Learning. In: International Conference on Machine Learning.Search in Google Scholar
15. Lunze, J. and B. Nixdorf. 2001. Representation of Hybrid Systems by Means of Stochastic Automata. Mathematical and Computer Modelling of Dynamical Systems 4(7): 383–422.10.1076/mcmd.7.4.383.3639Search in Google Scholar
16. Lunze, J. and J. Raisch. 2002. Discrete Models for Hybrid Systems. In: Modelling, Analysis, and Design of Hybrid Systems. Lecture Notes in Control and Information Sciences.Search in Google Scholar
17. Lunze, J. and J. Schröder. 2001. Computation of complete abstractions of quantised systems. In: European Control Conference.10.23919/ECC.2001.7076414Search in Google Scholar
18. Najafi, E., G. A. Lopes and R. Babuška. 2013. Reinforcement learning for sequential composition control. In: IEEE 52nd Annual Conference on Decision and Control (CDC), Florence, Italy.10.1109/CDC.2013.6761042Search in Google Scholar
19. Quah, T., D. Machalek and K. M. Powell. 2020. Comparing Reinforcement Learning Methods for Real-Time Optimization of a Chemical Process. Processes 8: 1497.10.3390/pr8111497Search in Google Scholar
20. Schwung, D., J. N. Reimann, A. Schwung and S. X. Ding. 2018. Self Learning in Flexible Manufacturing Units: A Reinforcement Learning Approach. In: International Conference on Intelligent Systems (IS 2018), Madeira, Portugal.10.1109/IS.2018.8710460Search in Google Scholar
21. Spielberg, S., A. Tulsyan, N. P. Lawrence, P. D. Loewen and B. Gopaluni. 2019. Toward self-driving processes: A deep reinforcement learning approach to control. AIChE Journal 65: e16689.10.1002/aic.16689Search in Google Scholar
22. Szita, I. and S. Csaba. 2010. Model-Based Reinforcement Learning with Nearly Tight Exploration Complexity Bounds. In ICML.Search in Google Scholar
23. Terzimehic, T., M. Wenger, A. Zoitl, A. Bayha, K. Becker, T. Müller and H. Schauerte. 2017. Towards an industry 4.0 compliant control software architecture using IEC 61499 and OPC UA. In: 22nd IEEE International Conference on Emerging Technologies and Factory Automation (ETFA).10.1109/ETFA.2017.8247718Search in Google Scholar
24. Wagner, C., C. v. Trotha, F. Palm and U. Epple. 2017. Fundamentals for the next Generation of Automation Solutions of the Fourth Industrial Revolution. In: The 2017 Asian Control Conference – ASCC 2017, Gold Coast, Australia.10.1109/ASCC.2017.8287596Search in Google Scholar
25. Yamasaki, T. and T. Ushio. 2005. Decentralized Supervisory Control of Discrete Event Systems Based on Reinforcement Learning. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences E88-A: 2982–2988.10.1093/ietfec/e88-a.11.2982Search in Google Scholar
26. Zhao, W., J. P. Queralta and T. Westerlund. 2020. Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey. In: IEEE Symposium Series on Computational Intelligence (SSCI).10.1109/SSCI47803.2020.9308468Search in Google Scholar
27. Zhu, L., Y. Cui, G. Takami H. Kanokogi and T. Matsubara. 2020. Scalable reinforcement learning for plant-wide control of vinyl acetate monomer process. Control Engineering Practice 97: 104331.10.1016/j.conengprac.2020.104331Search in Google Scholar
© 2022 Walter de Gruyter GmbH, Berlin/Boston