Abstract
Metal forming processes like open-die forging or hot rolling are well-established for the production of key components in various industries. Nevertheless, the control of the final microstructure and hence mechanical properties is not yet common. To achieve this, the authors propose and discuss a control concept based on reinforcement learning, fast process models (FPM) and an “operator in the loop” approach. The concept is explained and tested using deviating initial ingot temperatures as idealized process disruptions. RL algorithms are trained for both processes and transferred into the controllers that are connected to a simulative environment based on FPM. Within this framework, the online adaption is possible in ∼2 s in rolling and 4–6 s in forging. This highlights the concepts suitability to be used for property control in hot metal forming.
Zusammenfassung
Umformprozesse wie das Freiformschmieden oder das Walzen sind etablierte Verfahren zur Herstellung von Komponenten mit exzellenten Eigenschaften für verschiedenste Industriezweige. Trotzdem ist die Regelung finaler Werkstoffeigenschaften wie der Mikrostruktur noch nicht üblich. Um diesem Ziel näher zu kommen, stellen die Autoren ein Regelungskonzept vor, das auf Reinforcement Learning, schnellen Prozessmodellen (FPM) und einem “operator in the loop”-Ansatz basiert. Das Konzept wird erläutert und durch die Annahme der idealisierten Prozessstörung „abweichende Ausgangstemperatur im Werkstück“ getestet. RL-Algorithmen werden für beide Prozesse trainiert und in die Regler übertragen, die mit einer auf den FPM basierenden Simulationsumgebung verbunden sind. In diesem Rahmen ist die online Prozessadaption beim Walzen in ∼ 2 Sekunden und beim Schmieden in 4 bis 6 Sekunden möglich. Dies unterstreicht die Eignung des Konzepts für die Eigenschaftsregelung in der Warmumformung.
Funding source: Deutsche Forschungsgemeinschaft (DFG, German Research Foundation)
Award Identifier / Grant number: 452539735
Award Identifier / Grant number: 390621612
About the authors
Niklas Reinisch studierte Werkstoffingenieurwesen an der RWTH Aachen University und erhielt im Jahre 2020 seinen Masterabschluss. In seiner daran anschließenden Tätigkeit als Wissenschaftlicher Mitarbeiter und Gruppenleiter der Arbeitsgruppe Massivformteile beschäftigt er sich mit der optimierten Prozessauslegung im Freiformschmieden.
Christian Idzik studierte Automatisierungstechnik an der RWTH Aachen University und erhielt im Jahre 2018 seinen Masterabschluss. In seiner daran anschließenden Tätigkeit als Wissenschaftlicher Mitarbeiter in der Arbeitsgruppe Werkstoffmodellierung beschäftigt er sich mit der Digitalisierung und Optimierung des Warmwalzens.
David Bailly, geboren 1984, studierte Maschinenbau an der RWTH Aachen (Diplom 2010) und promovierte anschließend im Themenbereich Umformtechnik bei Prof. G. Hirt (Promotion 2016). Seit 2015 ist er Oberingenieur am Institut für Bildsame Formgebung an der RWTH Aachen. In seiner aktuellen Position als kommissarischer Leiter des Lehrstuhls für Umformtechnologien am IBF beschäftigt er sich mit verschiedenen Umformverfahren, deren Weiterentwicklung, Optimierung und Digitalisierung.
-
Research ethics: Not applicable.
-
Author contributions: The authors have accepted responsibility for the entire content of this manuscript and approved its submission.
-
Competing interests: The authors state no conflicts of interest.
-
Research funding: The work in the field of open die forging was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – 452539735. And the work in the field of rolling was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy – EXC-2023 Internet of Production – 390621612.
-
Data availability: Not applicable.
References
[1] J. M. Cullen, J. M. Allwood, and M. D. Bambach, “Mapping the global flow of steel: from steelmaking to end-use goods,” Environ. Sci. Technol., vol. 46, no. 24, pp. 13048–13055, 2012. https://doi.org/10.1021/es302433p.Search in Google Scholar PubMed
[2] J. M. Allwood, et al.., “Closed-loop control of product properties in metal forming,” CIRP Ann., vol. 65, no. 2, pp. 573–596, 2016. https://doi.org/10.1016/j.cirp.2016.06.002.Search in Google Scholar
[3] M. Wolfgarten and F. Knauf, “Advanced Pass-Schedule Design in Radial Forging,” [Online]. Available at: https://www.sms-group.com/de-de/insights/all-insights/advanced-pass-schedule-design-in-radial-forging Accessed: Dec. 11, 2023.Search in Google Scholar
[4] M. Wolfgarten, D. Rosenstock, F. Rudolph, and G. Hirt, “New approach for the optimization of pass-schedules in open-die forging,” Int. J. Mater. Form., vol. 12, no. 6, pp. 973–983, 2019. https://doi.org/10.1007/s12289-019-01471-w.Search in Google Scholar
[5] N. Chakraborti, B. Siva Kumar, V. Satish Babu, S. Moitra, and A. Mukhopadhyay, “A new multi-objective genetic algorithm applied to hot-rolling process,” Appl. Math. Model., vol. 32, no. 9, pp. 1781–1789, 2008. https://doi.org/10.1016/j.apm.2007.06.011.Search in Google Scholar
[6] N. Reinisch, D. Bailly, and G. Hirt, “Optimizing the microstructure in open-die forgings using reinforcement learning,” in 26th International ESAFORM Conference on Material Forming, 2023, pp. 2061–2070.10.21741/9781644902479-221Search in Google Scholar
[7] C. Idzik, A. Krämer, G. Hirt, and J. Lohmar, “Coupling of an analytical rolling model and reinforcement learning to design pass schedules: towards properties controlled hot rolling,” J. Intell. Manuf., 2023. https://doi.org/10.1007/s10845-023-02115-2.Search in Google Scholar
[8] J. Deng, S. Sierla, J. Sun, and V. Vyatkin, “Reinforcement learning for industrial process control: a case study in flatness control in steel industry,” Comput. Ind., vol. 143, p. 103748, 2022. https://doi.org/10.1016/j.compind.2022.103748.Search in Google Scholar
[9] D. Recker, M. Franzke, and G. Hirt, “Fast models for online-optimization during open die forging,” CIRP Ann., vol. 60, no. 1, pp. 295–298, 2011. https://doi.org/10.1016/j.cirp.2011.03.142.Search in Google Scholar
[10] N. Reinisch, F. Rudolph, S. Günther, D. Bailly, and G. Hirt, “Successful pass schedule design in open-die forging using double deep Q-learning,” Processes, vol. 9, no. 7, p. 1084, 2021. https://doi.org/10.3390/pr9071084.Search in Google Scholar
[11] C. Scheiderer, et al.., “Simulation-as-a-Service for reinforcement learning applications by example of heavy plate rolling processes,” Procedia Manuf., vol. 51, pp. 897–903, 2020. https://doi.org/10.1016/j.promfg.2020.10.126.Search in Google Scholar
[12] MINTEQ International GmbH, Ferrotron Division, LaCam® FORGE, Duisburg [Online], Available at: https://www.mineralstech.com/docs/default-source/refractories-documents/ferrotron/lacam-forge-brochure.pdf?sfvrsn=c58417e1_0 Accessed: Dec. 11, 2023.Search in Google Scholar
[13] F. Rudolph, M. Wolfgarten, V. Keray, and G. Hirt, “Optimization of open-die forging using fast models for strain, temperature, and grain size in the context of an assistance system,” in Forming the Future. The Minerals, Metals & Materials Series, G. Daehn, J. Cao, B. Kinsey, E. Tekkaya, A. Vivek, and Y. Yoshida, Eds., Cham, Springer, 2021, pp. 1145–1159.10.1007/978-3-030-75381-8_96Search in Google Scholar
[14] F. Y. Küçükakarsu, İ. İ. Ayhan, E. Alan, D. Taştemür, and S. Gündüz, “Effect of hot rolling process parameters on the microstructure and mechanical properties of continuously cooled low-carbon high-strength low-alloy (HSLA) steel,” Mater. Test., vol. 64, no. 8, pp. 1136–1149, 2022. https://doi.org/10.1515/mt-2021-2220.Search in Google Scholar
[15] R. Lafarge, S. Hütter, M. Tulke, T. Halle, and A. Brosius, “Data based model predictive control for ring rolling,” Prod. Eng. Res. Devel., vol. 15, no. 6, pp. 821–831, 2021. https://doi.org/10.1007/s11740-021-01063-1.Search in Google Scholar
[16] M. Bambach, M. Imran, I. Sizova, J. Buhl, S. Gerster, and M. Herty, “A soft sensor for property control in multi-stage hot forming based on a level set formulation of grain size evolution and machine learning,” Adv. Ind. Manuf. Eng., vol. 2, no. 3, p. 100041, 2021. https://doi.org/10.1016/j.aime.2021.100041.Search in Google Scholar
[17] M. Jiang, L.-N. Chen, J. He, G.-Y. Chen, C.-H. Li, and X.-G. Lu, “Effect of controlled rolling/controlled cooling parameters on microstructure and mechanical properties of the novel pipeline steel,” Adv. Manuf., vol. 2, no. 3, pp. 265–274, 2014. https://doi.org/10.1007/s40436-014-0084-z.Search in Google Scholar
[18] J. A. Polyblank, J. M. Allwood, and S. R. Duncan, “Closed-loop control of product properties in metal forming: a review and prospectus,” J. Mater. Process. Technol., vol. 214, no. 11, pp. 2333–2348, 2014. https://doi.org/10.1016/j.jmatprotec.2014.04.014.Search in Google Scholar
[19] D. Görges, “Relations between model predictive control and reinforcement learning,” IFAC-PapersOnLine, vol. 50, no. 1, pp. 4920–4928, 2017. https://doi.org/10.1016/j.ifacol.2017.08.747.Search in Google Scholar
[20] R. Nian, J. Liu, and B. Huang, “A review on reinforcement learning: introduction and applications in industrial process control,” Comput. Chem. Eng., vol. 139, p. 106886, 2020. https://doi.org/10.1016/j.compchemeng.2020.106886.Search in Google Scholar
[21] R. S. Sutton and A. Barto, Reinforcement Learning: An Introduction, Cambridge, MA, London, The MIT Press, 2018.Search in Google Scholar
[22] R. S. Sutton, A. Barto, and R. J. Williams, “Reinforcement learning is direct adaptive optimal control,” IEEE Control Syst., vol. 12, no. 2, pp. 19–22, 1992. https://doi.org/10.1109/37.126844.Search in Google Scholar
[23] D. Bertsekas, Reinforcement Learning and Optimal Control, Rowland Heights, CA, USA, Athena Scientific, 2019.Search in Google Scholar
[24] O. Gamal, M. I. P. Mohamed, C. G. Patel, and H. Roth, “Data-driven model-free intelligent roll gap control of bar and wire hot rolling process using reinforcement learning,” IJMERR, vol. 10, no. 7, pp. 349–356, 2021. https://doi.org/10.18178/ijmerr.10.7.349-356.Search in Google Scholar
[25] D. Rosenstock, et al.., “Online visualization during open die forging and optimization of pass schedules,” Steel Res. Int., vol. 85, no. 9, pp. 1348–1354, 2014. https://doi.org/10.1002/srin.201300396.Search in Google Scholar
[26] J. Lohmar, S. Seuren, M. Bambach, and G. Hirt, “Design and application of an advanced fast rolling model with through thickness resolution for heavy plate rolling,” in 2nd International Conference on Ingot Casting Rolling Forging: ICRF 2014, Milan, Italy, 2014.Search in Google Scholar
[27] A. Tomlinson and J. D. Stringer, “Spread and elongation in flat tool forging,” J. Iron Steel Inst., vol. 193, no. 2, pp. 157–162, 1959.Search in Google Scholar
[28] D. Ernst, M. Glavic, F. Capitanescu, and L. Wehenkel, “Reinforcement learning versus model predictive control: a comparison on a power system problem,” IEEE Trans Syst Man Cybern Part B Cybern, vol. 39, no. 2, pp. 517–529, 2009. https://doi.org/10.1109/TSMCB.2008.2007630.Search in Google Scholar PubMed
© 2024 Walter de Gruyter GmbH, Berlin/Boston