[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content

Fast and Robust Differential Private Stochastic Gradient Descent with Preconditioner

  • Conference paper
  • First Online:
Knowledge Management and Acquisition for Intelligent Systems (PKAW 2024)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 15372))

Included in the following conference series:

Abstract

Differential privacy (DP) is a widely used technique for securing the privacy of data in deep learning (DL). However, despite its broad adoption, the effectiveness of DP in safeguarding privacy has not yet been challenged. This paper introduces a novel approach, a fast and robust DP optimizer for general DL tasks, which addresses the limitations of existing methods and offers a fresh perspective on the issue. We first examine the convergence condition of current Differential Privacy (DP) algorithms. Then, we propose a new general optimization algorithm called ACNSI to reduce the effect of noise. ACNSI comprises five steps: affine transformation, clipping, noising, smoothing, and inverse affine transformation. By using affine and inverse affine transforms, we can significantly reduce the added Gaussian noise, which not only improves existing differential private stochastic gradient descent methods but also holds promise for enhancing the practical application of DP in DL tasks. We provide a theoretical analysis for privacy-preserving properties and utility improvement with noise reduction and smoothing effects. Finally, we conduct an empirical study to demonstrate that ACNSI achieves the best accuracy with a tight privacy budget and economical computational cost compared to existing methods.

Supported by the National Key R&D Program of China (No. 2023YFB2703700).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 39.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 49.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi, M., et al.: Deep learning with differential privacy. In: CCS, pp. 308–318 (2016)

    Google Scholar 

  2. Bassily, R., Feldman, V., Talwar, K., Thakurta, A.G.: Private stochastic convex optimization with optimal rates. In: NeurIPS, pp. 11282–11291 (2019)

    Google Scholar 

  3. Bassily, R., Smith, A., Thakurta, A.: Private empirical risk minimization: efficient algorithms and tight error bounds. In: FOCS, pp. 464–473. IEEE (2014)

    Google Scholar 

  4. Bertsekas, D.P., Tsitsiklis, J.N.: Parallel and Distributed Computation: Numerical Methods, vol. 23. Prentice hall Englewood Cliffs, NJ (1989)

    Google Scholar 

  5. Blocki, J., Blum, A., Datta, A., Sheffet, O.: The Johnson-lindenstrauss transform itself preserves differential privacy. In: 2012 IEEE 53rd Annual Symposium on Foundations of Computer Science, pp. 410–419. IEEE (2012)

    Google Scholar 

  6. Bu, Z., Dong, J., Long, Q., Su, W.J.: Deep learning with gaussian differential privacy. arXiv preprint arXiv:1911.11607 (2019)

  7. Chaudhuri, K., Monteleoni, C., Sarwate, A.D.: Differentially private empirical risk minimization. J. Mach. Learn. Res. 12(3), 1069–1109 (2011)

    MathSciNet  Google Scholar 

  8. Chen, L., Hu, X., Wu, H.: Randomized fast subspace descent methods. arXiv preprint arXiv:2006.06589 (2020)

  9. Chen, Y., Qin, X., Wang, J., Yu, C., Gao, W.: FedHealth: a federated transfer learning framework for wearable healthcare. IEEE Intell. Syst. 35, 83–93 (2020)

    Article  Google Scholar 

  10. Davody, A., Adelani, D.I., Kleinbauer, T., Klakow, D.: Robust differentially private training of deep neural networks. arXiv preprint arXiv:2006.10919 (2020)

  11. Dong, J., Roth, A., Su, W.J.: Gaussian differential privacy. arXiv preprint arXiv:1905.02383 (2019)

  12. Dwork, C., Roth, A., et al.: The algorithmic foundations of differential privacy. Found. Trends Theor. Comput. Sci. 9(3–4), 211–407 (2014)

    MathSciNet  Google Scholar 

  13. Geyer, R.C., Klein, T., Nabi, M.: Differentially private federated learning: a client level perspective. arXiv preprint arXiv:1712.07557 (2017)

  14. Haddadpour, F., Mahdavi, M.: On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425 (2019)

  15. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press (2012)

    Google Scholar 

  16. Jin, C., Ge, R., Netrapalli, P., Kakade, S.M., Jordan, M.I.: How to escape saddle points efficiently. In: ICML, pp. 1724–1732. PMLR (2017)

    Google Scholar 

  17. Mironov, I.: Rényi differential privacy. In: 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pp. 263–275. IEEE (2017)

    Google Scholar 

  18. Nesterov, Y.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    Article  MathSciNet  Google Scholar 

  19. Nikolov, A., Talwar, K., Zhang, L.: The geometry of differential privacy: the sparse and approximate cases. In: Proceedings of the Forty-fifth Annual ACM Symposium on Theory of Computing, pp. 351–360 (2013)

    Google Scholar 

  20. Osher, S., et al.: Laplacian smoothing gradient descent. arXiv preprint arXiv:1806.06317 (2018)

  21. Pichapati, V., Suresh, A.T., Yu, F.X., Reddi, S.J., Kumar, S.: AdaCliP: adaptive clipping for private SGD. arXiv preprint arXiv:1908.07643 (2019)

  22. Saad, Y.: Iterative Methods for Sparse Linear Systems. SIAM (2003)

    Google Scholar 

  23. Shang, F., Zhang, Z., Xu, T., Liu, Y., Liu, H.: Principal component analysis in the stochastic differential privacy model. In: UAI, pp. 1110–1119. PMLR (2021)

    Google Scholar 

  24. Shen, D., Wu, G., Suk, H.I.: Deep learning in medical image analysis. Annu. Rev. Biomed. Eng. 19, 221–248 (2017)

    Article  Google Scholar 

  25. Shokri, R., Stronati, M., Song, C., Shmatikov, V.: Membership inference attacks against machine learning models. In: 2017 IEEE Symposium on Security and Privacy (SP), pp. 3–18. IEEE (2017)

    Google Scholar 

  26. Upadhyay, J.: Randomness efficient fast-Johnson-lindenstrauss transform with applications in differential privacy and compressed sensing. arXiv preprint arXiv:1410.2470 (2014)

  27. Wang, B., Gu, Q., Boedihardjo, M., Wang, L., Barekat, F., Osher, S.J.: DP-LSSGD: a stochastic optimization method to lift the utility in privacy-preserving ERM. In: Mathematical and Scientific Machine Learning, pp. 328–351. PMLR (2020)

    Google Scholar 

  28. Wang, D., Ye, M., Xu, J.: Differentially private empirical risk minimization revisited: faster and more general. arXiv preprint arXiv:1802.05251 (2018)

  29. Wright, J., Ma, Y.: High-Dimensional Data Analysis with Low-Dimensional Models: Principles, Computation, and Applications. Cambridge University Press (2022)

    Google Scholar 

  30. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Huiwen Wu .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wu, H. (2025). Fast and Robust Differential Private Stochastic Gradient Descent with Preconditioner. In: Wu, S., Su, X., Xu, X., Kang, B.H. (eds) Knowledge Management and Acquisition for Intelligent Systems. PKAW 2024. Lecture Notes in Computer Science(), vol 15372. Springer, Singapore. https://doi.org/10.1007/978-981-96-0026-7_15

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-0026-7_15

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-0025-0

  • Online ISBN: 978-981-96-0026-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics