More Web Proxy on the site http://driver.im/

research-article

Architectural patterns for the design of federated learning systems

Authors:

Hye-Young Paik,

Chen WangAuthors Info & Claims

Volume 191, Issue C

https://doi.org/10.1016/j.jss.2022.111357

Published: 01 September 2022 Publication History

Abstract

Federated learning has received fast-growing interests from academia and industry to tackle the challenges of data hungriness and privacy in machine learning. A federated learning system can be viewed as a large-scale distributed system with different components and stakeholders as numerous client devices participate in federated learning. Designing a federated learning system requires software system design thinking apart from the machine learning knowledge. Although much effort has been put into federated learning from the machine learning technique aspects, the software architecture design concerns in building federated learning systems have been largely ignored. Therefore, in this paper, we present a collection of architectural patterns to deal with the design challenges of federated learning systems. Architectural patterns present reusable solutions to a commonly occurring problem within a given context during software architecture design. The presented patterns are based on the results of a systematic literature review and include three client management patterns, four model management patterns, three model training patterns, four model aggregation patterns, and one configuration pattern. The patterns are associated to the particular state transitions in a federated learning model lifecycle, serving as a guidance for effective use of the patterns in the design of federated learning systems.

Highlights

•

Federated learning systems as large-scale distributed systems with different components and stakeholders requires software system design thinking.

•

A collection of architectural patterns based on a systematic literature review to support the federated learning system design.

•

15 architectural patterns: 3 client management patterns, 4 model management patterns, 3 model training patterns, 4 model aggregation patterns, and 1 configuration pattern.

•

Comprehensive and systematic derivation and description of the pattern languages for all 15 architectural patterns.

References

[1]

Ahmad A., Jamshidi P., Pahl C., Khaliq F., A pattern language for the evolution of component-based software architectures, ECEASST 59 (2014).

[2]

Ahn J., Simeone O., Kang J., Wireless federated distillation for distributed edge learning with heterogeneous data, in: PIMRC 2019, 2019, pp. 1–6.

[3]

Bai, J., Zhou, K., Xue, G., Zha, H., Sun, G., Tseng, B., Zheng, Z., Chang, Y., 2009. Multi-task learning for learning to rank in web search. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management, pp. 1549–1552.

[4]

Balalaie A., Heydarnoori A., Jamshidi P., Tamburri D.A., Lynn T., Microservices migration patterns, Softw. - Pract. Exp. 48 (11) (2018) 2019–2042.

[5]

Bao X., Su C., Xiong Y., Huang W., Hu Y., Flchain: A blockchain for auditable federated learning with trust and incentive, in: BIGCOM ’19, 2019, pp. 151–159.

[6]

Beck K., Cunningham W., Using pattern languages for object oriented programs, in: OOPSLA ’12, 1987.

[7]

Bonawitz K., Eichner H., Grieskamp W., Huba D., Ingerman A., Ivanov V., Kiddon C., Konečný J., Mazzocchi S., McMahan B., Van Overveldt T., Petrou D., Ramage D., Roselander J., Towards federated learning at scale: system design, Proceedings of Machine Learning and Systems, 2019, pp. 374–388. https://proceedings.mlsys.org/paper/2019/file/bd686fd640be98efaae0091fa301e613-Paper.pdf.

[8]

Bonawitz K., Ivanov V., Kreuter B., Marcedone A., McMahan H.B., Patel S., Ramage D., Segal A., Seth K., Practical Secure Aggregation for Privacy-Preserving Machine Learning, Association for Computing Machinery, New York, NY, USA, 2017.

[9]

Bowden G.J., Maier H.R., Dandy G.C., Real-time deployment of artificial neural network forecasting models: Understanding the range of applicability, Water Resour. Res. 48 (10) (2012).

[10]

Briggs, C., Fan, Z., Andras, P., 2020. Federated learning with hierarchical clustering of local updates to improve training on non-IID data. In: 2020 International Joint Conference on Neural Networks (IJCNN). pp. 1–9. https://doi.org/10.1109/IJCNN48605.2020.9207469.

[11]

Brinkkemper S., Method engineering: engineering of information systems development methods and tools, Inf. Softw. Technol. 38 (4) (1996) 275–280. Method Engineering and Meta-Modelling.

[12]

Chai Z., Ali A., Zawad S., Truex S., Anwar A., Baracaldo N., Zhou Y., Ludwig H., Yan F., Cheng Y., Tifl: A tier-based federated learning system, in: ACM HPDC, in: HPDC ’20, Association for Computing Machinery, New York, NY, USA, 2020, pp. 125–136.

[13]

Chen Y., Ning Y., Slawski M., Rangwala H., Asynchronous online federated learning for edge devices with non-iid data, 2020 IEEE International Conference on Big Data (Big Data), IEEE Computer Society, Los Alamitos, CA, USA, 2020, pp. 15–24,. https://doi.ieeecomputersociety.org/10.1109/BigData50022.2020.9378161.

[14]

Corinzia L., Buhmann J.M., Variational federated multi-task learning, 2019, arXiv:1906.06268.

[15]

Duan M., Liu D., Chen X., Tan Y., Ren J., Qiao L., Liang L., Astraea: Self-balancing federated learning for improving classification accuracy of mobile deep learning applications, in: ICCD 2019, 2019, pp. 246–254.

[16]

EU M., General data protection regulation (GDPR), 2020, URL https://eur-lex.europa.eu/eli/reg/2016/679/oj.

[17]

Gu B., Xu A., Huo Z., Deng C., Huang H., Privacy-preserving asynchronous federated learning algorithms for multi-party vertically collaborative learning, 2020, arXiv:2008.06233.

[18]

Haddadpour F., Kamani M.M., Mokhtari A., Mahdavi M., Federated learning with compression: unified analysis and sharp guarantees, International Conference on Artificial Intelligence and Statistics, PMLR, 2021, pp. 2350–2358.

[19]

Hu C., Jiang J., Wang Z., Decentralized federated learning: A segmented gossip approach, 2019, arXiv:1908.07782.

[20]

Huang L., Shea A.L., Qian H., Masurkar A., Deng H., Liu D., Patient clustering improves efficiency of federated machine learning to predict mortality and hospital stay time using distributed electronic medical records, J. Biomed. Inform. 99 (2019).

Digital Library

[21]

Jamshidi P., Pahl C., Mendonça N.C., Pattern-based multi-cloud architecture migration, Softw. - Pract. Exp. 47 (9) (2017) 1159–1184.

[22]

Jeong E., Oh S., Kim H., Park J., Bennis M., Kim S.-L., Communication-efficient on-device machine learning: Federated distillation and augmentation under non-IID private data, 2018, arXiv:1811.11479.

[23]

Jiang J., Hu L., Decentralised federated learning with adaptive partial gradient aggregation, CAAI Trans. Intell. Technol. 5 (3) (2020) 230–236.

[24]

Jiang Y., Wang S., Valls V., Ko B.J., Lee W.-H., Leung K.K., Tassiulas L., Model pruning enables efficient federated learning on edge devices, IEEE Trans. Neural Netw. Learn. Syst. (2022) 1–13,.

[25]

Jobin A., Ienca M., Vayena E., The global landscape of AI ethics guidelines, Nat. Mach. Intell. 1 (9) (2019) 389–399.

[26]

Kitchenham B., Charters S., Guidelines for performing systematic literature reviews in software engineering, 2007.

Digital Library

[27]

Konečný J., McMahan H.B., Yu F.X., Richtárik P., Suresh A.T., Bacon D., Federated learning: Strategies for improving communication efficiency, 2017, arXiv:1610.05492.

[28]

Lakshmanan L.V., Munn M., Robinson S., Machine Learning Design Patterns, 2020.

[29]

Lalitha, A., Shekhar, S., Javidi, T., Koushanfar, F., 2018. Fully decentralized federated learning. In: Third Workshop on Bayesian Deep Learning (NeurIPS).

[30]

Li T., Sahu A.K., Talwalkar A., Smith V., Federated learning: Challenges, methods, and future directions, IEEE Signal Process. Mag. 37 (3) (2020) 50–60.

[31]

Liu Y., Ai Z., Sun S., Zhang S., Liu Z., Yu H., Yang Q., Fan L., Yu H. (Eds.), FedCoin: A Peer-to-Peer Payment System for Federated Learning, Springer International Publishing, Cham, 2020, pp. 125–138.

[32]

Liu L., Zhang J., Song S.H., Letaief K.B., Client-edge-cloud hierarchical federated learning, in: ICC 2020, 2020, pp. 1–6.

[33]

Lo S.K., Liew C.S., Tey K.S., Mekhilef S., An interoperable component-based architecture for data-driven IoT system, Sensors 19 (20) (2019).

[34]

Lo S.K., Lu Q., Paik H.-Y., Zhu L., FLRA: A reference architecture for federated learning systems, in: Software Architecture, Springer International Publishing, 2021, pp. 83–98.

[35]

Lo S.K., Lu Q., Wang C., Paik H.-Y., Zhu L., A systematic literature review on federated machine learning: From a software engineering perspective, ACM Comput. Surv. 54 (5) (2021).

Digital Library

[36]

Lwakatare L.E., Raj A., Bosch J., Olsson H.H., Crnkovic I., A taxonomy of software engineering challenges for machine learning systems: An empirical investigation, in: Kruchten P., Fraser S., Coallier F. (Eds.), Agile Processes in Software Engineering and Extreme Programming, Springer International Publishing, Cham, 2019, pp. 227–243.

[37]

McMahan B., Moore E., Ramage D., Hampson S., y Arcas B.A., Communication-efficient learning of deep networks from decentralized data, Artificial intelligence and statistics, PMLR, 2017, pp. 1273–1282.

[38]

Meszaros G., Doble J., A pattern language for pattern writing, in: Pattern Languages of Program Design 3, Addison-Wesley Longman Publishing Co., Inc., USA, 1997, pp. 529–574.

[39]

Mills J., Hu J., Min G., Communication-efficient federated learning for wireless edge intelligence in IoT, IEEE Internet Things J. 7 (7) (2020) 5986–5994.

[40]

Mirbel I., Ralyté J., Situational method engineering: combining assembly-based and roadmap-driven approaches, Requir. Eng. 11 (1) (2006) 58–78.

[41]

Roy A.G., Siddiqui S., Pölsterl S., Navab N., Wachinger C., Braintorrent: A peer-to-peer environment for decentralized federated learning, 2019, arXiv:1905.06731.

[42]

Sattler F., Wiedemann S., Müller K.R., Samek W., Robust and communication-efficient federated learning from non-i.i.d. data, IEEE Trans. Neural Netw. Learn. Syst. 31 (9) (2020) 3400–3413.

[43]

Smith, V., Chiang, C.-K., Sanjabi, M., Talwalkar, A., 2017. Federated multi-task learning. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. pp. 4427–4437.

[44]

Wan Z., Xia X., Lo D., Murphy G.C., How does machine learning change software development practices?, IEEE Trans. Softw. Eng. (2019) 1.

[45]

Wang L., Wang W., Li B., CMFL: Mitigating communication overhead for federated learning, in: ICDCS 2019, 2019, pp. 954–964.

[46]

Warnat-Herresthal S., Schultze H., Shastry K.L., Manamohan S., Mukherjee S., Garg V., Sarveswara R., Händler K., Pickkers P., Aziz N.A., et al., Swarm learning for decentralized and confidential clinical machine learning, Nature 594 (7862) (2021) 265–270.

[47]

Washizaki, H., Uchida, H., Khomh, F., Guéhéneuc, Y., 2019. Studying software engineering patterns for designing machine learning systems. In: 2019 10th International Workshop on Empirical Software Engineering in Practice (IWESEP), pp. 49–495.

[48]

Weng J., Weng J., Zhang J., Li M., Zhang Y., Luo W., Deepchain: Auditable and privacy-preserving deep learning with blockchain-based incentive, IEEE Trans. Dependable Secure Comput. (2019) 1.

[49]

Xie C., Koyejo S., Gupta I., Asynchronous federated optimization, 2020, arXiv:1903.03934.

[50]

Xu R., Baracaldo N., Zhou Y., Anwar A., Ludwig H., Hybridalpha: An efficient approach for privacy-preserving federated learning, in: AISec’19, Association for Computing Machinery, New York, NY, USA, 2019, pp. 13–23.

[51]

Xu, Z., Yu, F., Xiong, J., Chen, X., 2021. Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration. In: 2021 58th ACM/IEEE Design Automation Conference (DAC). pp. 997–1002. https://doi.org/10.1109/DAC18074.2021.9586241.

[52]

Yang Q., Liu Y., Chen T., Tong Y., Federated machine learning: Concept and applications, ACM Trans. Intell. Syst. Technol. 10 (2) (2019).

Digital Library

[53]

Yokoyama H., Machine learning system architectural pattern for improving operational stability, in: ICSA-C 2019, 2019, pp. 267–274.

[54]

Zdun U., Systematic pattern selection using pattern language grammars and design space analysis, Softw. - Pract. Exp. 37 (9) (2007) 983–1016.

[55]

Zhang W., Lu Q., Yu Q., Li Z., Liu Y., Lo S.K., Chen S., Xu X., Zhu L., Blockchain-based federated learning for device failure detection in industrial IoT, IEEE Internet Things J. 8 (7) (2021) 5926–5937,.

Cited By

Lu QZhu LXu XWhittle JZowghi DJacquet A(2024)Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and EngineeringACM Computing Surveys10.1145/362623456:7(1-35)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3626234
Aggarwal MKhullar VGoyal NProla T(2024)Resource-efficient federated learning over IoAT for rice leaf disease classificationComputers and Electronics in Agriculture10.1016/j.compag.2024.109001221:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.compag.2024.109001
Wang ZLi H(2023)An Efficient Federated Learning Method Based on Optimized-residual and ClusteringProceedings of the 15th International Conference on Digital Image Processing10.1145/3604078.3604153(1-7)Online publication date: 19-May-2023
https://dl.acm.org/doi/10.1145/3604078.3604153
Show More Cited By

Index Terms

Architectural patterns for the design of federated learning systems
1. Computer systems organization
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language features
        Patterns
  2. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Client-server architectures
      2. Software architectures

Index terms have been assigned to the content through auto-classification.

Recommendations

FLRA: A Reference Architecture for Federated Learning Systems
Software Architecture
Abstract
Federated learning is an emerging machine learning paradigm that enables multiple devices to train models locally and formulate a global model, without sharing the clients’ local data. A federated learning system can be viewed as a large-scale ...
Multimodal federated learning: Concept, methods, applications and future directions
Abstract
Multimodal learning mines and analyzes multimodal data in reality to better understand and appreciate the world around people. However, how to exploit this rich multimodal data without violating user privacy is a key issue. Federated learning is ...
Highlights
- The three different modes in the multimodal federated learning model are summarized.
- Multimodal fusion based on the federated learning framework is also specified.
- The difficulties and some ideas of multimodal federated learning ...
Performance Analysis of Distributed and Federated Learning Models on Private Data
Abstract
There has been significant research in privacy-related aspects of machine learning and large scale data processing. In traditional methods of training a model, data is gathered at a centralized machine where training on the entire data takes ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Systems and Software

Journal of Systems and Software Volume 191, Issue C

Sep 2022

181 pages

ISSN:0164-1212

Issue’s Table of Contents

Elsevier Inc.

Publisher

Elsevier Science Inc.

United States

Publication History

Published: 01 September 2022

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lu QZhu LXu XWhittle JZowghi DJacquet A(2024)Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and EngineeringACM Computing Surveys10.1145/362623456:7(1-35)Online publication date: 9-Apr-2024
https://dl.acm.org/doi/10.1145/3626234
Aggarwal MKhullar VGoyal NProla T(2024)Resource-efficient federated learning over IoAT for rice leaf disease classificationComputers and Electronics in Agriculture10.1016/j.compag.2024.109001221:COnline publication date: 18-Jul-2024
https://dl.acm.org/doi/10.1016/j.compag.2024.109001
Wang ZLi H(2023)An Efficient Federated Learning Method Based on Optimized-residual and ClusteringProceedings of the 15th International Conference on Digital Image Processing10.1145/3604078.3604153(1-7)Online publication date: 19-May-2023
https://dl.acm.org/doi/10.1145/3604078.3604153
Wang YLi J(2023)Anomaly Detection Method for Time Series Data Based on Transformer ReconstructionProceedings of the 2023 12th International Conference on Informatics, Environment, Energy and Applications10.1145/3594692.3594702(58-63)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3594692.3594702
Smestad CLi J(2023)A Systematic Literature Review on Client Selection in Federated LearningProceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering10.1145/3593434.3593438(2-11)Online publication date: 14-Jun-2023
https://dl.acm.org/doi/10.1145/3593434.3593438

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents