[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content

Advertisement

Springer Nature Link
Log in

Designing parallel data processing for enabling large-scale sensor applications

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Masses of sensors are being deployed at the scale of cities to manage parking spaces, transportation infrastructures to monitor traffic, and campuses of buildings to reduce energy consumption. These large-scale infrastructures become a reality for citizens via applications that orchestrate sensors to deliver high-value, innovative services. These applications critically rely on the processing of large amounts of data to analyze situations, inform users, and control devices. This paper proposes a design-driven approach to developing orchestrating applications for masses of sensors that integrates parallel processing of large amounts of data. Specifically, an application design exposes declarations that are used to generate a programming framework based on the MapReduce programming model. We have developed a prototype of our approach, using Apache Hadoop. We applied it to a case study and obtained significant speedups by parallelizing computations over twelve nodes. In doing so, we demonstrate that our design-driven approach allows to abstract over implementation details, while exposing architectural properties used to generate high-performance code for processing large datasets. Furthermore, we show that this high-performance support enables new, personalized services in a smart city. Finally, we discuss the expressiveness of our design language, identify some limitations, and present language extensions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. http://phoenix.inria.fr/software/diaswarm.

  2. http://eclipse.org/.

  3. http://hadoop.apache.org/.

  4. http://phoenix.inria.fr/software/diaswarm.

  5. https://www.waze.com.

  6. https://www.google.com/maps/about/.

  7. http://phoenix.inria.fr/software/diaswarm.

References

  1. Garcia-Ayllon S, Miralles JL (2015) New strategies to improve governance in territorial management: evolving from smart cities to smart territories. Procedia Eng 118:3–11

    Article  Google Scholar 

  2. Mizuno Y, Odake N (2015) Current status of smart systems and case studies of privacy protection platform for smart city in Japan. In: 2015 Portland international conference on management of engineering and technology (PICMET), pp 612–624

  3. Naphade M, Banavar G, Harrison C, Paraszczak J, Morris R (2011) Smarter cities and their innovation challenges. Computer 44(6):32–39

    Article  Google Scholar 

  4. Libelium (2016) Smart City project in Santander to monitor Parking Free Slots [Online]. http://www.libelium.com/smart_santander_parking_smart_city

  5. Worldsensing SL (2016) Worldsensing and SIGFOX announce the world’s largest Intelligent Parking deployment with Micronet, the SIGFOX Network Operator for Russia [Online]. http://www.worldsensing.com/news-press/press-release-worldsensing-and-sigfox-announce-the-worlds-largest-intelligent-parking-deployment-with-micronet-the-sigfox-network-operator-for-russia.html

  6. Lee K-H, Lee Y-J, Choi H, Chung YD, Moon B (2012) Parallel data processing with MapReduce: a survey. SIGMOD Rec 40(4):11–20

    Article  Google Scholar 

  7. Patsakis C, Venanzi R, Bellavista P, Solanas A, Bouroche M (2014) Personalized medical services using smart cities’ infrastructures. In: Proceeding of the 2014 IEEE international symposium on medical measurements and applications (MeMeA). IEEE, pp 665–669

  8. Seeliger R, Krauss C, Wilson A, Zwicklbauer M, Arbanowski S (2015) Towards personalized smart city guide services in future internet environments. In: Proceedings of the 24th international conference on World Wide Web, ser. WWW ’15 companion. ACM, New York, pp 563–568 [Online]. doi:10.1145/2740908.2743905

  9. The Apache Software Foundation (2016) Apache Pig [Online]. https://pig.apache.org

  10. The Apache Software Foundation (2016) Apache Hive [Online]. https://hive.apache.org

  11. Chambers C, Raniwala A, Perry F, Adams S, Henry RR, Bradshaw R, Weizenbaum N (2010) FlumeJava: easy, efficient data-parallel pipelines. In: Proceedings of the 31st ACM SIGPLAN conference on programming language design and implementation (PLDI ’10). ACM, pp 363–375

  12. Kabáč M, Consel C (2015) Orchestrating masses of sensors: a design-driven development approach. In: Proceedings of the 2015 ACM SIGPLAN international conference on generative programming: concepts and experiences (GPCE 2015). ACM, pp 117–120

  13. Kabáč M, Consel C (2016) Designing parallel data processing for large-scale sensor orchestration. In: 13th IEEE international conference on ubiquitous intelligence and computing (UIC 2016), Toulouse, France, best Paper Award [Online]. https://hal.inria.fr/hal-01319730

  14. Lämmel R (2008) Google’s MapReduce programming model—revisited. Sci Comput Program 70(1):1–30

    Article  MathSciNet  MATH  Google Scholar 

  15. Dean J, Ghemawat S (2008) MapReduce: simplified data processing on large clusters. Commun ACM 51(1):107–113

    Article  Google Scholar 

  16. Taylor RN, Medvidovic N, Dashofy EM (2009) Software architecture: foundations, theory, and practice. Wiley, New York

    Book  Google Scholar 

  17. Fayad M, Schmidt DC (1997) Object-oriented application frameworks. Commun ACM 40(10):32–38

    Article  Google Scholar 

  18. Tilak S, Abu-Ghazaleh NB, Heinzelman W (2002) A taxonomy of wireless micro-sensor network models. SIGMOBILE Mob Comput Commun Rev 6(2):28–36

    Article  Google Scholar 

  19. The Apache Software Foundation (2016) Hadoop Wiki PoweredBy [Online]. http://wiki.apache.org/hadoop/PoweredBy

  20. White T (2012) Hadoop: the definitive guide. O’Reilly Media Inc, Sebastopol

    Google Scholar 

  21. The Apache Software Foundation (2016) Projects directory [Online]. https://projects.apache.org/projects.html?category#big-data

  22. The Apache Software Foundation (2016) Apache Spark [Online]. http://spark.apache.org

  23. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX conference on hot topics in cloud computing (HotCloud’10). USENIX Association, pp 10–10

  24. Zaharia M, Chowdhury M, Das T, Dave A, Ma J, McCauley M, Franklin MJ, Shenker S, Stoica I (2012) Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on networked systems design and implementation (NSDI’12). USENIX Association, p 2

  25. The Apache Software Foundation (2016) Apache Storm [Online]. http://storm.apache.org

  26. Cugola G, Margara A (2012) Processing flows of information: from data stream to complex event processing. ACM Comput Surv 44(3):15:1–15:62

    Article  Google Scholar 

  27. Hewlett-Packard (2016) HP Helion eucalyptus [Online]. http://www.eucalyptus.com

  28. DataStax (2016) DataStax enterprise [Online]. http://www.datastax.com

  29. Mackowski D, Bai Y, Ouyang Y (2015) Parking space management via dynamic performance-based pricing. Transp Res Procedia 7:170–191

    Article  Google Scholar 

  30. Pfrommer J, Warrington J, Schildbach G, Morari M (2014) Dynamic vehicle redistribution and online price incentives in shared mobility systems. IEEE Trans Intell Transp Syst 15(4):1567–1578

    Article  Google Scholar 

  31. Kabáč M, Consel C, Volanschi N (2016) Leveraging declarations over the lifecycle of large-scale sensor applications. In: 13th IEEE international conference on ubiquitous intelligence and computing (UIC 2016). Toulouse, France [Online]. https://hal.inria.fr/hal-01319731

  32. Patel P, Pathak A, Cassou D, Issarny V (2013) Enabling high-level application development in the internet of things. In: S-CUBE’13: 4th international conference on sensor systems and software

  33. Serral E, Valderas P, Pelechano V (2010) Towards the model driven development of context-aware pervasive systems. Pervasive Mob Comput 6(2):254–280

    Article  Google Scholar 

  34. Dey AK, Abowd GD, Salber D (2001) A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Hum Comput Interact 16(2):97–166

    Article  Google Scholar 

  35. Ranganathan A, Chetan S, Al-Muhtadi J, Campbell RH, Mickunas MD (2005) Olympus: a high-level programming model for pervasive computing environments. In: Proceedings of the third IEEE international conference on pervasive computing and communications (PERCOM ’05). IEEE Computer Society, pp 7–16

  36. Cassou D, Balland E, Consel C, Lawall J (2011) Leveraging software architectures to guide and verify the development of sense/compute/control applications. In: Proceedings of the 33rd international conference on software engineering (ICSE ’11). ACM, pp 431–440

  37. Bertran B, Bruneau J, Cassou D, Loriant N, Balland E, Consel C (2014) DiaSuite: a tool suite to develop sense/compute/control applications. Sci Comput Program 79:39–51

    Article  Google Scholar 

  38. Gupta V, Tovar E, Pinho LM, Kim J, Lakshmanan K, Rajkumar R (2011) sMapReduce: a programming pattern for wireless sensor networks. In: Proceedings of the 2nd workshop on software engineering for sensor network applications (SESENA ’11). ACM, pp 37–42

  39. Mottola L, Picco GP (2011) Programming wireless sensor networks: fundamental concepts and state of the art. ACM Comput Surv 43(3):19:1–19:51

    Article  Google Scholar 

  40. Pike R, Dorward S, Griesemer R, Quinlan S (2005) Interpreting the data: parallel analysis with Sawzall. Sci Program 13(4):277–298

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Inria Bordeaux, Bordeaux, France

    Milan Kabáč & Nic Volanschi

  2. Bordeaux Institute of Technology, Bordeaux, France

    Charles Consel

Authors
  1. Milan Kabáč
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Charles Consel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nic Volanschi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nic Volanschi.

Appendix: DiaSwarm grammar

Appendix: DiaSwarm grammar

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kabáč, M., Consel, C. & Volanschi, N. Designing parallel data processing for enabling large-scale sensor applications. Pers Ubiquit Comput 21, 457–473 (2017). https://doi.org/10.1007/s00779-017-1009-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-017-1009-1

Keywords

Search

Navigation