Point and Control it!: Using Computer Vision for Service Discovery to Control Smart Objects
Pages 153 - 160
Abstract
The widespread of smart objects in our daily lives request the creation and analysis of new service discovery mechanisms and interaction techniques. In this work, we designed and evaluated a pointing-based interaction mechanism based on a Convolutional Neural Network classification method. We called it ViSCo (View, Scan, and Control it), which extends the openHAB service discovery mechanism of smart objets. ViSCo aggregates the users' field of view, captured by the camera of their smartphones, to reduce the service discovery results. 17 users evaluated the final solution remotely, in an environment with virtual devices. Participants used the ViSCo approach to find and control virtual devices by pointing to real objects in their homes (e.g., their TVs). System Usability Scale (SUS) survey about ViSCo results showed a good level of acceptance, with an average score of 83.97.
References
[1]
M. Aziez, S. Benharzallah, and H. Bennoui. 2017. Service discovery for the Internet of Things: Comparison study of the approaches. In 2017 4th International Conference on Control, Decision and Information Technologies (CoDIT). IEEE, Barcelona, Spain, 0599--0604. https://doi.org/10.1109/CoDIT.2017.8102660
[2]
Meriem Aziez, Saber Benharzallah, and Hammadi Bennoui. 2019. A full comparison study of service discovery approaches for internet of things. International Journal of Pervasive Computing and Communications 15 (2019), 30--56.
[3]
Senthil Murugan Balakrishnan and Arun Kumar Sangaiah. 2015. Aspect oriented middleware for Internet of things: a state-of-the art survey of service discovery approaches. Int. J. Intell. Eng. Syst 8, 4 (2015), 16--28.
[4]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114--123.
[5]
Vincent Becker, Felix Rauchenstein, and Gábor Sörös. 2020. Connecting and Controlling Appliances Through Wearable Augmented Reality. Augmented Human Research 5, 1 (2020), 1--16.
[6]
Christian Bettstetter and Christoph Renner. 2000. A comparison of service discovery protocols and implementation of the service location protocol. In Proceedings of the 6th EUNICE Open European Summer School: Innovative Internet Applications. CTIT/UT, Enschede, the Netherlands, 101--108.
[7]
Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. 2020. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020).
[8]
Christian Cabrera, Andrei Palade, and Siobhán Clarke. 2017. An Evaluation of Service Discovery Protocols in the Internet of Things. In Proceedings of the Symposium on Applied Computing (Marrakech, Morocco) (SAC '17). Association for Computing Machinery, New York, NY, USA, 469--476. https://doi.org/10.1145/3019612.3019698
[9]
Kaifei Chen, Jonathan Fürst, John Kolb, Hyung-Sin Kim, Xin Jin, David E. Culler, and Randy H. Katz. 2018. SnapLink: Fast and Accurate Vision-Based Appliance Control in Large Commercial Buildings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 129 (Jan. 2018), 27 pages.
[10]
Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Shlomi Regev, et al. 2020. Tensorflow lite micro: Embedded machine learning on tinyml systems. arXiv preprint arXiv:2010.08678 (2020).
[11]
Adrian A. de Freitas, Michael Nebeling, Xiang 'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K. Dey. 2016. Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). ACM, New York, NY, USA, 5909--5920.
[12]
eclipse SmartHome. 2019. Eclipse SmartHome - A Flexible Framework for the Smart Home. https://www.eclipse.org/smarthome/getting-started.html. (Accessed on 05/28/2019).
[13]
J. A. Garcia Macias, J. Alvarez-Lozano, P. Estrada, and E. Aviles Lopez. 2011. Browsing the Internet of Things with Sentient Visors. Computer 44, 5 (May 2011), 46--52. https://doi.org/10.1109/MC.2011.128
[14]
Dominique Guinard, Vlad Trifa, Stamatis Karnouskos, Patrik Spiess, and Domnic Savio. 2010. Interacting with the soa-based internet of things: Discovery, query, selection, and on-demand provisioning of web services. IEEE transactions on Services Computing 3, 3 (2010), 223--235.
[15]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017).
[16]
S. Kajwadkar and V. K. Jain. 2018. ECTX-CoAP: An Enhanced Context-Aware CoAP Extension for Smart Objects Discovery in Internet of Things. In 2018 Conference on Information and Communication Technology (CICT). 1--6.
[17]
Jung-Hwa Kim, Seung-June Choi, and Jin-Woo Jeong. 2019. Watch & Do: A smart iot interaction system with object detection and gaze estimation. IEEE Transactions on Consumer Electronics 65, 2 (2019), 195--204.
[18]
James R Lewis. 2018. The system usability scale: past, present, and future. International Journal of Human-Computer Interaction 34, 7 (2018), 577--590.
[19]
Jerome Luckenbac, David Gräff, Georgios Moutsos, Christoph Weitkamp, and Miika Jukka. 2020. Bindings | openHAB. https://www.openhab.org/docs/developer/bindings/. (Accessed on 02/16/2020).
[20]
S. Mayer and J. Siegel. 2015. Conversations with connected vehicles. In 2015 5th International Conference on the Internet of Things (IOT). 38--44.
[21]
S. Mayer and G. Sörös. 2014. User Interface Beaming - Seamless Interaction with Smart Things Using Personal Wearable Computers. In 2014 11th International Conference on Wearable and Implantable Body Sensor Networks Workshops. IEEE, 46--49. https://doi.org/10.1109/BSN.Workshops.2014.17
[22]
Jakob Nielsen. 2000. Why you only need to test with 5 users.
[23]
Thomas P Novak and Donna L Hoffman. 2019. Relationship journeys in the internet of things: a new framework for understanding interactions between consumers and smart objects. Journal of the Academy of Marketing Science 47, 2 (2019), 216--237.
[24]
A. Pacheco, E. Flores, R. Sánchez, and S. Almanza-García. 2018. Smart Classrooms Aided by Deep Neural Networks Inference on Mobile Devices. In 2018 IEEE International Conference on Electro/Information Technology (EIT). IEEE, https://ieeexplore.ieee.org/abstract/document/8500260, 0605--0609.
[25]
Enrico Rukzio, Gregor Broll, Karin Leichtenstern, and Albrecht Schmidt. 2007. Mobile interaction with the real world: An evaluation and comparison of physical mobile interaction techniques. In European Conference on Ambient Intelligence. Springer, 1--18.
[26]
Julian Seifert, Andreas Bayer, and Enrico Rukzio. 2013. PointerPhone: Using Mobile Phones for Direct Pointing Interactions with Remote Displays. In Human-Computer Interaction - INTERACT 2013, Paula Kotzé, Gary Marsden, Gitte Lindgaard, Janet Wesson, and Marco Winckler (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 18--35.
[27]
Romulo Silva, Windson Viana, and Paulo Filipe Dantas. 2018. Using images to extend smart object discovery in an Internet of Things scenario. In Anais Estendidos do XXIV Simpósio Brasileiro de Sistemas Multimídia e Web. SBC, 57--60.
[28]
TensorFlow. 2021. TensorFlow Lite converter. https://www.tensorflow.org/lite/convert/ (Accessed on 20/04/2021).
[29]
Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in statistics. Springer, 196--202.
[30]
WordNet. 2020. WordNet | A Lexical Database for English. https://wordnet.princeton.edu/. (Accessed on 03/19/2020).
[31]
Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 4000--4008. https://doi.org/10.1145/3025453.3025828
Index Terms
- Point and Control it!: Using Computer Vision for Service Discovery to Control Smart Objects
Recommendations
Enablement of IoT Based Context-Aware Smart Home with Fog Computing
The recent advent of Internet of Things IoT, has given rise to a plethora of smart verticals-smart homes being one of them. Smart Home is a classic example of IoT, wherein smart appliances connected via home gateways constitute a local home network to ...
Smart Meter for Smart Homes: Data Mining Applications of Produced Big Data
BDE '19: Proceedings of the 2019 International Conference on Big Data EngineeringWe attempt to address the opportunities spawned from the voluminous data generated by the incorporation of smart technology within a building. Furthermore, we suggest a framework to discover the acceptance pattern of smart technologies. In the absence of ...
Internet of Things for enabling smart environments: A technology-centric perspective
The Internet of Things (IoT) is a computing paradigm whereby everyday life objects are augmented with computational and wireless communication capabilities, typically through the incorporation of resource-constrained devices including sensors and ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
November 2021
271 pages
ISBN:9781450386098
DOI:10.1145/3470482
- General Chairs:
- Adriano César Machado Pereira,
- Leonardo Chaves Dutra da Rocha
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
In-Cooperation
- SBC: Brazilian Computer Society
- CNPq: Conselho Nacional de Desenvolvimento Cientifico e Tecn
- CAPES: Brazilian Higher Education Funding Council
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 November 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
WebMedia '21: Brazilian Symposium on Multimedia and the Web
November 5 - 12, 2021
Minas Gerais, Belo Horizonte, Brazil
Acceptance Rates
WebMedia '21 Paper Acceptance Rate 24 of 75 submissions, 32%;
Overall Acceptance Rate 270 of 873 submissions, 31%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 62Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)0
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in