More Web Proxy on the site http://driver.im/

research-article

Open access

Federated Fine-Tuning of LLMs on the Very Edge: The Good, the Bad, the Ugly

Authors:

Herbert Woisetschläger,

Alexander Erben,

Hans-Arno JacobsenAuthors Info & Claims

DEEM '24: Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning

Pages 39 - 50

https://doi.org/10.1145/3650203.3663331

Published: 09 June 2024 Publication History

Abstract

With the emergence of AI regulations, such as the EU AI Act, requirements for simple data lineage, enforcement of low data bias, and energy efficiency have become a priority for everyone offering AI services. Being pre-trained on versatile and a vast amount of data, large language models and foundation models (FMs) offer a good basis for building high-quality deep learning pipelines. Fine-tuning can further improve model performance on a specific downstream task, which requires orders of magnitude less data than pre-training. Often, access to high-quality and low-bias data for model fine-tuning is limited due to technical or regulatory requirements. Federated learning (FL), as a distributed and privacy-preserving technique, offers a well-suited approach to significantly expanding data access for model fine-tuning. Yet, this data is often located on the network edge, where energy, computational, and communication resources are significantly more limited than in data centers.

In our paper, we conduct an end-to-end evaluation for fine-tuning the FLAN-T5 FM family on the network edge. We study energy efficiency potentials throughout FL systems - on clients, in communication, and on the server. Our analysis introduces energy efficiency as a real-time metric to assess the computational efficiency of an FL system. We show the stark need for further improvements in communication efficiency when working with FMs and demonstrate the importance of adaptive FL optimizers for FM training.

References

[1]

3GPP. 4g (lte advanced). https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2585, 2008.

[2]

Amazon Web Services (AWS). Amazon AWS p3 instance types. https://aws.amazon.com/ec2/instance-types/p3/, 2023. Accessed: 2023-09-27.

[3]

Sara Babakniya, Ahmed Elkordy, et al. SLoRA: Federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023. URL https://openreview.net/forum?id=06quMTmtRV.

[4]

Sebastian Baunsgaard, Matthias Boehm, and et al. Exdra: Exploratory data science on federated raw data. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD/PODS '21. ACM, June 2021. URL http://dx.doi.org/10.1145/3448016.3457549.

Digital Library

[5]

Daniel J. Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque de Gusmão, and Nicholas D. Lane. Flower: A friendly federated learning research framework, 2020. URL https://arxiv.org/abs/2007.14390.

[6]

Rishi Bommasani, Drew A. Hudson, et al. On the opportunities and risks of foundation models. ArXiv, 2021. URL https://crfm.stanford.edu/assets/report.pdf.

[7]

Sebastian Caldas, Peter Wu, Tian Li, Jakub Konecný, H. Brendan McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: A benchmark for federated settings. CoRR, abs/1812.01097, 2018. URL http://arxiv.org/abs/1812.01097.

[8]

Aakanksha Chowdhery, Sharan Narang, et al. Palm: scaling language modeling with pathways. J. Mach. Learn. Res., 24(1), mar 2024. ISSN 1532-4435.

[9]

Hyung Won Chung, Le Hou, et al. Scaling instruction-finetuned language models. 2022. URL https://arxiv.org/abs/2210.11416.

[10]

Council of the European Union. Proposal for a REGULATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL - LAYING DOWN HARMONISED RULES ON ARTIFICIAL INTELLIGENCE (ARTIFICIAL INTELLIGENCE ACT) AND AMENDING CERTAIN UNION LEGISLATIVE ACTS, apr 2021. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206. Document 52021PC0206.

[11]

Radosvet Desislavov, Fernando Martínez-Plumed, and José Hernández-Orallo. Trends in ai inference energy consumption: Beyond the performance-vs-parameter laws of deep learning. Sustainable Computing: Informatics and Systems, 38:100857, 2023. ISSN 2210-5379. URL https://www.sciencedirect.com/science/article/pii/S2210537923000124.

[12]

Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. Fate-llm: A industrial grade federated learning framework for large language models, 2023. URL https://arxiv.org/abs/2310.10049.

[13]

Jie Feng, Lei Liu, Qingqi Pei, and Keqin Li. Min-max cost optimization for efficient hierarchical federated learning in wireless edge networks. IEEE Transactions on Parallel and Distributed Systems, 33(11):2687--2700, 2021.

[14]

Nathan C. Frey, Baolin Li, Joseph McDonald, Dan Zhao, Michael Jones, David Bestor, Devesh Tiwari, Vijay Gadepally, and Siddharth Samsi. Benchmarking resource usage for efficient distributed deep learning. In 2022 IEEE High Performance Extreme Computing Conference (HPEC), pages 1--8, 2022.

[15]

Xue-Yong Fu, Md Tahmid Rahman Laskar, et al. Tiny titans: Can smaller large language models punch above their weight in the real world for meeting summarization?, 2024. URL https://arxiv.org/abs/2402.00841.

[16]

Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70--79, Hong Kong, China, November 2019. Association for Computational Linguistics. URL https://aclanthology.org/D19-5409.

[17]

Chaoyang He, Songze Li, et al. Fedml: A research library and benchmark for federated machine learning, 2020. URL https://arxiv.org/abs/2007.13518.

[18]

Jordan Hoffmann, Sebastian Borgeaud, et al. An empirical analysis of compute-optimal large language model training. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=iBBcRUlOAPR.

[19]

House Of Commons of Canada. An Act to enact the Consumer Privacy Protection Act, the Personal Information and Data Protection Tribunal Act and the Artificial Intelligence and Data Act and to make consequential and related amendments to other Acts, 6 2022. URL https://www.parl.ca/DocumentViewer/en/44-1/bill/C-27/first-reading.

[20]

Edward J Hu, Yelong Shen, et al. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.

[21]

Kai Hwang. Advanced Computer Architecture: Parallelism,Scalability,Programmability. McGraw-Hill Higher Education, 1st edition, 1992. ISBN 0070316228.

[22]

Andrei Ivanov, Nikoli Dryden, Tal Ben-Nun, Shigang Li, and Torsten Hoefler. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3:711--732, 2021.

[23]

Fatemeh Jalali, Rob Ayre, Arun Vishwanath, Kerry Hinton, Tansu Alpcan, and Rod Tucker. Energy consumption of content distribution from nano data centers versus centralized data centers. ACM SIGMETRICS Performance Evaluation Review, 42(3):49--54, December 2014. ISSN 0163-5999. URL http://dx.doi.org/10.1145/2695533.2695555.

Digital Library

[24]

Jiayin Jin, Jiaxiang Ren, Yang Zhou, Lingjuan Lyu, Ji Liu, and Dejing Dou. Accelerated federated learning with decoupled adaptive optimization. 2022. URL https://arxiv.org/abs/2207.07223.

[25]

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.

[26]

Kacper Kubiak, Grzegorz Dec, and Dorota Stadnicka. Possible applications of edge computing in the manufacturing industry---systematic literature review. Sensors, 22(7):2445, March 2022. ISSN 1424-8220. URL http://dx.doi.org/10.3390/s22072445.

[27]

Fan Lai, Yinwei Dai, Sanjay S. Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, and Mosharaf Chowdhury. FedScale: Benchmarking model and system performance of federated learning at scale. In International Conference on Machine Learning (ICML), 2022.

[28]

Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013.

[29]

Wei Liu, Li Chen, et al. Accelerating federated learning via momentum gradient descent. IEEE Transactions on Parallel and Distributed Systems, 31(8):1754--1766, 2020.

[30]

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.

[31]

Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, and Furu Wei. The era of 1-bit llms: All large language models are in 1.58 bits, 2024. URL https://arxiv.org/abs/2402.17764.

[32]

Grigory Malinovsky, Samuel Horváth, Konstantin Pavlovich Burlachenko, and Peter Richtárik. Federated learning with regularized client participation. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and Opportunities, 2023. URL https://openreview.net/forum?id=6CDBpf7kNG.

[33]

Brendan McMahan, Eider Moore, et al. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273--1282. PMLR, 20-22 Apr 2017. URL https://proceedings.mlr.press/v54/mcmahan17a.html.

[34]

Samuel S. Ogden and Tian Guo. MODI: Mobile deep inference made efficient by edge computing. In USENIX Workshop on Hot Topics in Edge Computing (HotEdge 18), Boston, MA, July 2018. USENIX Association. URL https://www.usenix.org/conference/hotedge18/presentation/ogden.

[35]

Zhen Qin, Daoyuan Chen, et al. Federated full-parameter tuning of billion-sized language models with communication cost under 18 kilobytes, 2023. URL https://arxiv.org/abs/2312.06353.

[36]

Sashank J. Reddi, Zachary Charles, et al. Adaptive federated optimization. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=LkFG3lB13U5.

[37]

Jie Ren, Samyam Rajbhandari, et al. Zero-offload: Democratizing billion-scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551--564, 2021.

[38]

Swapnil Sadashiv Shinde, Arash Bozorgchenani, Daniele Tarchi, and Qiang Ni. On the design of federated learning in latency and energy constrained computation offloading operations in vehicular edge computing systems. IEEE Transactions on Vehicular Technology, 71(2): 2041--2057, 2021.

[39]

Yuanyishu Tian, Yao Wan, Lingjuan Lyu, Dezhong Yao, Hai Jin, and Lichao Sun. FedBERT: When federated learning meets pre-training. ACM Transactions on Intelligent Systems and Technology, 13(4):1--26, August 2022. URL https://doi.org/10.1145/3510033.

Digital Library

[40]

Martino Trevisan, Ali Safari Khatouni, and Danilo Giordano. Errant: Realistic emulation of radio access networks. Computer Networks, 176: 107289, July 2020. ISSN 1389-1286. URL http://dx.doi.org/10.1016/j.comnet.2020.107289.

[41]

Blesson Varghese, Nan Wang, David Bermbach, Cheol-Ho Hong, Eyal De Lara, Weisong Shi, and Christopher Stewart. A survey on edge performance benchmarking. ACM Comput. Surv., 54(3), apr 2021. ISSN 0360-0300. URL https://doi.org/10.1145/3444692.

Digital Library

[42]

Xiaofei Wang, Yiwen Han, Chenyang Wang, Qiyang Zhao, Xu Chen, and Min Chen. In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning. Ieee Network, 33 (5):156--165, 2019.

Digital Library

[43]

Herbert Woisetschläger, Alexander Erben, Bill Marino, Shiqiang Wang, Nicholas D. Lane, Ruben Mayer, and Hans-Arno Jacobsen. Federated learning priorities under the european union artificial intelligence act, 2024. URL https://arxiv.org/abs/2402.05968.

[44]

Mengwei Xu, Dongqi Cai, Yaozong Wu, Xiang Li, and Shangguang Wang. Fwdllm: Efficient fedllm using forward gradient, 2023. URL https://arxiv.org/abs/2308.13894.

[45]

Yunfan Ye, Shen Li, Fang Liu, Yonghao Tang, and Wanting Hu. Edgefed: Optimized federated learning based on edge computing. IEEE Access, 8:209191--209198, 2020. ISSN 2169-3536. URL http://dx.doi.org/10.1109/ACCESS.2020.3038287.

[46]

Ashkan Yousefpour, Shen Guo, Ashish Shenoy, Sayan Ghosh, Pierre Stock, Kiwan Maeng, Schalk-Willem Krüger, Michael Rabbat, Carole-Jean Wu, and Ilya Mironov. Green federated learning, 2023. URL https://arxiv.org/abs/2303.14604.

[47]

Rong Yu and Peichun Li. Toward resource-efficient federated learning in mobile edge computing. IEEE Network, 35(1):148--155, 2021.

Digital Library

[48]

Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedGPT: Federated instruction tuning. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023. URL https://openreview.net/forum?id=TaDiklyVps.

[49]

Zhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang, Yue Yu, Lizhen Qu, and Zenglin Xu. FedPETuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 9963--9977, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.findings-acl.632.

[50]

Yanli Zhao, Andrew Gu, et al. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16(12):3848--3860, aug 2023. ISSN 2150-8097. URL https://doi.org/10.14778/3611540.3611569.

Digital Library

[51]

Jingjing Zheng, Kai Li, Eduardo Tovar, and Mohsen Guizani. Federated learning for energy-balanced client selection in mobile edge computing. In 2021 International Wireless Communications and Mobile Computing (IWCMC), pages 1942--1947. IEEE, 2021.

Cited By

Piccialli FChiaro DQi PBellandi VDamiani E(2025)Federated and edge learning for large language modelsInformation Fusion10.1016/j.inffus.2024.102840117(102840)Online publication date: May-2025
https://doi.org/10.1016/j.inffus.2024.102840
Mao YYu XHuang KAngela Zhang YZhang J(2024)Green Edge AI: A Contemporary SurveyProceedings of the IEEE10.1109/JPROC.2024.3437365112:7(880-911)Online publication date: Jul-2024
https://doi.org/10.1109/JPROC.2024.3437365

Recommendations

Poster: Patching NSEC3-Encloser: The Good, the Bad, and the Ugly
CCS '24: Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security

This paper evaluates the effectiveness of patches designed to mitigate the NSEC3-encloser attack in DNS resolvers. NSEC3, used in DNSSEC to authenticate non-existence of records, can be exploited to exhaust resolver resources through excessive SHA-1 ...
An FPGA-based Fine Tuning Accelerator for a Sparse CNN
FPGA '19: Proceedings of the 2019 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays

Fine-tuning learns abundant feature expression for a wide range of natural images by using a pre-trained CNN model. It can be applied to a wide range of the neural network (NN)based computer vision problems. This paper proposes an FPGA-based fine-tuning ...
MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning
KDD '24: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

Code LLMs have emerged as a specialized research field, with remarkable studies dedicated to enhancing model's coding capabilities through fine-tuning on pre-trained models. Previous fine-tuning approaches were typically tailored to specific downstream ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

DEEM '24: Proceedings of the Eighth Workshop on Data Management for End-to-End Machine Learning

June 2024

89 pages

ISBN:9798400706110

DOI:10.1145/3650203

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 June 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

German Federal Ministry for Economic Affairs and Climate Action
German Research Foundation
Bavarian Ministry of Economic Affairs, Regional Development and Energy

Conference

SIGMOD/PODS '24

Sponsor:

SIGMOD

SIGMOD/PODS '24: International Conference on Management of Data

June 9, 2024

AA, Santiago, Chile

Acceptance Rates

DEEM '24 Paper Acceptance Rate 12 of 17 submissions, 71%;

Overall Acceptance Rate 44 of 67 submissions, 66%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
629
Total Downloads

Downloads (Last 12 months)629
Downloads (Last 6 weeks)136

Reflects downloads up to 17 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Piccialli FChiaro DQi PBellandi VDamiani E(2025)Federated and edge learning for large language modelsInformation Fusion10.1016/j.inffus.2024.102840117(102840)Online publication date: May-2025
https://doi.org/10.1016/j.inffus.2024.102840
Mao YYu XHuang KAngela Zhang YZhang J(2024)Green Edge AI: A Contemporary SurveyProceedings of the IEEE10.1109/JPROC.2024.3437365112:7(880-911)Online publication date: Jul-2024
https://doi.org/10.1109/JPROC.2024.3437365

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents