More Web Proxy on the site http://driver.im/

research-article

A Multiplexing Scheme for Multimodal Teleoperation

Authors:

Rahul Chaudhari,

Christoph Bachhuber,

Eckehard SteinbachAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 13, Issue 2

Article No.: 21, Pages 1 - 28

https://doi.org/10.1145/3063594

Published: 11 April 2017 Publication History

Abstract

This article proposes an application-layer multiplexing scheme for teleoperation systems with multimodal feedback (video, audio, and haptics). The available transmission resources are carefully allocated to avoid delay-jitter for the haptic signal potentially caused by the size and arrival time of the video and audio data. The multiplexing scheme gives high priority to the haptic signal and applies a preemptive-resume scheduling strategy to stream the audio and video data. The proposed approach estimates the available transmission rate in real time and adapts the video bitrate, data throughput, and force buffer size accordingly. Furthermore, the proposed scheme detects sudden transmission rate drops and applies congestion control to avoid abrupt delay increases and converge promptly to the altered transmission rate. The performance of the proposed scheme is measured objectively in terms of end-to-end signal latencies, packet rates, and peak signal-to-noise ratio (PSNR) for visual quality. Moreover, peak-delay and convergence time measurements are carried out to investigate the performance of the congestion control mode of the system.

Supplementary Material

cizmeci (cizmeci.zip)

Supplemental movie, appendix, image and software files for, A Multiplexing Scheme for Multimodal Teleoperation

Download
47.04 MB

References

[1]

Allied Vision. 1992. MAKO G-223 Gigabit Ethernet Industrial Machine Vision Camera. Retrieved March 1, 2017 from https://www.alliedvision.com.

[2]

APPOSITE-TECH. 2005. Netropy N60 Hardware Network Emulator. Retrieved March 1, 2017 from http://www.apposite-tech.com/.

[3]

C. Bachhuber and E. Steinbach. 2016. A system for high precision glass-to-glass delay measurements in video communication. In IEEE International Conference on Image Processing (ICIP’16). 2132--2136.

[4]

L. S. Brakmo and L. L. Peterson. 1995. TCP Vegas: End-to-end congestion avoidance on a global internet. IEEE Journal on Selected Areas in Communications 13, 1465--1480.

Digital Library

[5]

G. C. Burdea. 1996. Force and Touch Feedback for Virtual Reality. John Wiley 8 Sons, New York, NY.

Digital Library

[6]

A. Capone, L. Fratta, and F. Martignon. 2004. Bandwidth estimation schemes for TCP over wireless networks. IEEE Transactions on Mobile Computing 3, 2, 129--143.

Digital Library

[7]

C. Casetti, M. Gerla, S. Mascolo, M. Y. Sanadidi, and R. Wang. 2002. TCP westwood: End-to-end congestion control for wired/wireless networks. Springer-Verlag Wireless Networks 8, 5, 467--479.

Digital Library

[8]

Z. Cen, M. Mutka, Y. Liu, A. Goradia, and N. Xi. 2005a. QoS management of supermedia enhanced teleoperation via overlay networks. In IEEE International Conference on Intelligent Robots and Systems. 1630--1635.

[9]

Z. Cen, M. W. Mutka, Danyu Zhu, and Ning Xi. 2005b. Improved transport service for remote sensing and control over wireless networks. In IEEE International Conference on Mobile Adhoc and Sensor Systems. 340--347.

[10]

J. Cha, Y. Seo, Y. Kim, and J. Ryu. 2007. An authoring/editing framework for haptic broadcasting: Passive haptic interactions using MPEG-4 BIFS. In IEEE Proceedings of EuroHaptics Conference. 274--279.

Digital Library

[11]

B. Cizmeci, R. Chaudhari, X. Xu, N. Alt, and E. Steinbach. 2014. A visual-haptic multiplexing scheme for teleoperation over constant-bitrate communication links. Haptics: Neuroscience, Devices, Modeling, and Applications, Lecture Notes in Computer Science, Vol. 8619, 131--138.

[12]

J. E. Colgate and J. M. Brown. 1994. Factors affecting the z-width of a haptic display. In IEEE International Conference on Robotics and Automation. 3205--3210.

[13]

M. U. Demircin, P. van Beek, and Y. Altunbasak. 2008. Delay-constrained and r-d optimized transrating for high-definition video streaming over WLANs. IEEE Transactions on Multimedia 10, 6, 1155--1168.

Digital Library

[14]

E. Dinc and O. B. Akan. 2015. More than the eye can see: Coherence time and coherence bandwidth of troposcatter links for mobile receivers. IEEE Vehicular Technology Magazine 10, 2, 86--92.

[15]

M. A. Eid, J. Cha, and A. El-Saddik. 2011. Admux: An adaptive multiplexer for haptic-audio-visual data communication. IEEE Transactions on Instrumentation and Measurement 60, 1, 21--31.

[16]

W. R. Ferrell. 1965. Remote manipulation with transmission delay. IEEE Transactions on Human Factors in Electronics 6, 1, 24--32.

[17]

FORCE DIMENSION. 2001. Omega 6, 6-DoF Haptic Device. Retrieved March 1, 2017 from http://www.forcedimension.com/.

[18]

M. Gao, B. Cizmeci, M. Eiler, E. Steinbach, D. Zhao, and W. Gao. 2015. Macroblock level rate control for low delay H.264/AVC based video communication. In IEEE Picture Coding Symposium. 210--215.

[19]

M. Goeller, J. Oberlaender, K. Uhl, A. Roennau, and R. Dillmann. 2012. Modular robots for on-orbit satellite servicing. In IEEE International Conference on Robotics and Biomimetics (ROBIO’12). 2018--2023.

[20]

P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss. 2008. Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems. IEEE Transactions on Signal Processing 56, 2, 588--597.

Digital Library

[21]

P. Hinterseer, E. Steinbach, S. Hirche, and M. Buss. 2005. A novel psychophysically motivated transmission approach for haptic data streams in telepresence and teleaction systems. In IEEE International Conference on Acoustics, Speech and Signal Processing. 1097--1100.

[22]

S. Hirche, P. Hinterseer, E. Steinbach, and M. Buss. 2005. Network traffic reduction in haptic telepresence systems by deadband control. In 16th IFAC World Congress. 77--82.

[23]

E. Isomura, S. Tasaka, and T. Nunome. 2011. QoE enhancement in audiovisual and haptic interactive IP communications by media adaptive intra-stream synchronization. In TENCON 2011 - IEEE Region 10 Conference. 1085--1089.

[24]

ITU-T. 1993. ITU-T JPEG Standard, Digital Compression and Coding of Continuous-tone Still Images.

[25]

ITU-T. 2005. ITU-T H.264, Advanced Video Coding for Generic Audiovisual Services.

[26]

S. Kaede, T. Nunome, and S. Tasaka. 2015. QoE enhancement of audiovisual and haptic interactive IP communications by user-assistance. In IEEE International Conference on Computational Science and Engineering. 35--42.

Digital Library

[27]

KUKA Robotics. 1898. KUKA Leight Weight Robot Arm. (1898). Retrieved March 1, 2017 from http://www.kuka-robotics.com/.

[28]

M. Kuschel, P. Kremer, S. Hirche, and M. Buss. 2006. Lossy data reduction methods for haptic telepresence systems. In IEEE International Conference on Robotics and Automation. 2933--2938.

[29]

S. Lee, S. Moon, and J. W. Kim. 2006. A network-adaptive transport scheme for haptic-based collaborative virtual environments. In Proceedings of 5th ACM SIGCOMM Workshop on Network and System Support for Games, Article No. 13.

Digital Library

[30]

Alan Marshall, Kian Meng Yap, and Wai Yu. 2008. Providing QoS for networked peers in distributed haptic virtual environments. Hindawi, Journal of Advances in Multimedia 2008, 841590.

[31]

L. Merritt and R. Vanam. 2004. x264: A High Performance H.264/AVC Encoder. Retrieved March 1, 2017 from http://www.videolan.org/developers/x264.html.

[32]

H. A. Osman, M. Eid, R. Iglesias, and A. El Saddik. 2007. ALPHAN: Application layer protocol for haptic networking. In IEEE International Workshop on Haptic, Audio and Visual Environments and Games. 96--101.

[33]

L. Ping, L. Wenjuan, and S. Zengqi. 2005. Transport layer protocol reconfiguration for network-based robot control system. In IEEE Proceedings of Networking, Sensing and Control. 1049--1053.

[34]

PortAudio. 1999. Portable Cross Platform Audio I/O. Retrieved March 1, 2017 from http://portaudio.com/.

[35]

J. Pyke, M. Hart, V. Popov, R. D. Harris, and S. McGrath. 2007. A tele-ultrasound system for real-time medical imaging in resource-limited settings. In IEEE International Conference of Engineering in Medicine and Biology Society. 3094--3097.

[36]

John Ramming. 1983. JR3 6-DoF Force-Torque Sensor. Retrieved March 1, 2017 from http://www.jr3.com/.

[37]

M. Rank, Z. Shi, H. Müller, and S. Hirche. 2010. Perception of delay in haptic telepresence systems. MIT Presence v.19, n.5 (2010), p. 389--399.

Digital Library

[38]

J.-H. Ryu, J. Artigas, and C. Preusche. 2010. A passive bilateral control scheme for a teleoperator with time-varying communication delay. Mechatronics 20, 7, 812--823.

[39]

T. B. Sheridan. 1993. Space teleoperation through time delay: Review and prognosis. IEEE Transactions on Robotics and Automation 9, 5, 592--606.

[40]

J. M. Silva, M. Orozco, J. Cha, A. El Saddik, and E. M. Petriu. 2013. Human perception of haptic-to-video and haptic-to-audio skew in multimedia applications. ACM Transactions on Multimedia Computing, Communications and Applications 9, 2, 9:1--9:16.

Digital Library

[41]

I. Stoica, S. Shenker, and H. Zhang. 1998. Core-stateless fair queueing: Achieving approximately fair bandwidth allocations in high speed networks. ACM SIGCOMM Computer Communication Review 28, 4, 118--130.

Digital Library

[42]

Y. Uchimura, K. Ohnishi, and T. Yakoh. 2002. Bilateral robot system on the real time network structure. In IEEE International Workshop on Advanced Motion Control. 63--68.

[43]

J. M. Valin, T. B. Terriberry, C. Montgomery, and G. Maxwell. 2010. A high-quality speech and audio codec with less than 10-ms delay. IEEE Transactions on Audio, Speech, and Language Processing 18, 1, 58--67.

Digital Library

[44]

I. Vogels. 2004. Detection of temporal delays in visual-haptic interfaces. HUMAN FACTORS 46, 1, 118--134.

[45]

J. Walraevens. 2005. Discrete-Time Queueing Models with Priorities. Ph.D. Dissertation, (TELIN) Ghent University, Ghent, Belgium.

[46]

E. Weber. 1851. Die lehre vom tastsinn und gemeingefuehl, auf versuche gegruendet. Vieweg: Braunschweig, Germany.

[47]

X. Xu, B. Cizmeci, C. Schuwerk, and E. Steinbach. 2015. Haptic data reduction for time-delayed teleoperation using the time domain passivity approach. In IEEE World Haptics Conference. 512--518.

[48]

S. Yamamoto, D. Yashiro, K. Yubai, and S. Komada. 2014. Rate control based on queuing state observer for visual-haptic communication. In IEEE International Workshop on Advanced Motion Control. 569--574.

[49]

D. Yashiro, D. Tian, and T. Yakoh. 2013. End-to-end flow control for visual-haptic communication in the presence of bandwidth change. Wiley Electronics and Communications in Japan 96, 11, 26--34.

[50]

Youtube. 2016a. Combining Haptic Data Reduction with Stabilizing Control Approaches. Retrieved March 1, 2017 from https://youtu.be/0GzIm2R8ICY.

[51]

Youtube. 2016b. Ultra-Low Delay Video Transmission. Retrieved March 1, 2017 from https://youtu.be/dl9Figh-ymk.

[52]

Youtube. 2017. A Multiplexing Scheme for Multimodal Teleoperation. Retrieved March 1, 2017 from https://youtu.be/a1MSrvENElg.

[53]

F. Zhang and E. Steinbach. 2011. Improved ρ-domain rate control with accurate header size estimation. In IEEE International Conference on Acoustics, Speech and Signal Processing. 813--816.

[54]

H. Zhihai and S. K. Mitra. 2002. Optimum bit allocation and accurate rate control for video coding via ρ-domain source modeling. IEEE Transactions on Circuits and Systems for Video Technology 12, 10, 840--849.

Digital Library

Cited By

S. PK. S. P(2024)Enhanced Light-Gradient Boosting Machine (GBM)-Based Artificial Intelligence-Blockchain-Based Telesurgery in Sixth Generation Communication Using Optimization ConceptJournal of Electrical and Computer Engineering10.1155/2024/32145722024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/3214572
Chen YLi AWu DZhou L(2024)Toward General Cross-Modal Signal Reconstruction for Robotic TeleoperationIEEE Transactions on Multimedia10.1109/TMM.2023.331294426(3541-3553)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3312944
Kroep KGokhale VVerburg JPrasad R(2024) ETVO: Effectively Measuring Tactile Internet With Experimental Validation IEEE Transactions on Mobile Computing10.1109/TMC.2023.324665923:3(2054-2065)Online publication date: Mar-2024
https://doi.org/10.1109/TMC.2023.3246659
Show More Cited By

Index Terms

A Multiplexing Scheme for Multimodal Teleoperation

Recommendations

Efficient multiplexing scheme for IP packets over the advanced satellite broadcasting system

Implementing an IP-friendly transport mechanism in broadcasting systems would enable more convenient multimedia broadcasting applications. The overhead of current multiplexing methods for IP packets that are based on MPEG-2 systems is large, and they ...
Performance Analysis of Reassembly and Multiplexing Queueing with Long-Range-Dependent Input Traffic

This paper studies the impact of long-range-dependent (LRD) traffic on the performance of reassembly and multiplexing queueing. A queueing model characterizing the general reassembly and multiplexing operations performed in packet networks is developed ...
An Enhanced TCP Scheme for Distinguishing Non-congestion Losses from Packet Reordering over Wireless Mesh Networks
HPCC '11: Proceedings of the 2011 IEEE International Conference on High Performance Computing and Communications

Wireless Mesh Network (WMN) is a communication networks, which aims to provide ubiquitous high bandwidth access for a large number of users. Transmission Control Protocol (TCP) is the most popular connection oriented transport layer protocol used in the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 13, Issue 2

May 2017

226 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3058792

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 April 2017

Accepted: 01 March 2017

Revised: 01 March 2017

Received: 01 October 2016

Published in TOMM Volume 13, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

European Union's Seventh Framework Programme (FP7/2007-2013)/ERC
European Research Council

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
492
Total Downloads

Downloads (Last 12 months)78
Downloads (Last 6 weeks)11

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

S. PK. S. P(2024)Enhanced Light-Gradient Boosting Machine (GBM)-Based Artificial Intelligence-Blockchain-Based Telesurgery in Sixth Generation Communication Using Optimization ConceptJournal of Electrical and Computer Engineering10.1155/2024/32145722024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/3214572
Chen YLi AWu DZhou L(2024)Toward General Cross-Modal Signal Reconstruction for Robotic TeleoperationIEEE Transactions on Multimedia10.1109/TMM.2023.331294426(3541-3553)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3312944
Kroep KGokhale VVerburg JPrasad R(2024) ETVO: Effectively Measuring Tactile Internet With Experimental Validation IEEE Transactions on Mobile Computing10.1109/TMC.2023.324665923:3(2054-2065)Online publication date: Mar-2024
https://doi.org/10.1109/TMC.2023.3246659
Wei XLiao JZhou LSari HZhuang W(2024)Toward Generic Cross-Modal Transmission StrategyIEEE Transactions on Communications10.1109/TCOMM.2024.339280172:10(6059-6072)Online publication date: Oct-2024
https://doi.org/10.1109/TCOMM.2024.3392801
Wu DZhou L(2024)Cross-Modal Stream Transmission: Architecture, Strategy, and TechnologyIEEE Wireless Communications10.1109/MWC.013.220029331:1(134-140)Online publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1109/MWC.013.2200293
Tong QWei WLiu CZhou XZhang YWang D(2024)Cross-Modal Transmission with Active Packet Loss and Restoration for Tactile InternetIEEE Communications Magazine10.1109/MCOM.002.230070262:8(70-76)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/MCOM.002.2300702
Yuan ZWei XZhou LZhuang W(2024)Content-Aware Cross-Modal Stream TransmissionIEEE Wireless Communications Letters10.1109/LWC.2024.342284113:9(2507-2511)Online publication date: Sep-2024
https://doi.org/10.1109/LWC.2024.3422841
Ma MYang YDoan TLhamo OFitzek FNguyen G(2024)SynCoDel: Network-Assisted Synchronization of Video and Haptic Streams for Teleoperations2024 IEEE 49th Conference on Local Computer Networks (LCN)10.1109/LCN60385.2024.10639694(1-6)Online publication date: 8-Oct-2024
https://doi.org/10.1109/LCN60385.2024.10639694
Kroep KGokhale VSimha APrasad RRao VMitra SVenkatasubramanian NDubey AFeng LGhasemi MSprinkle J(2023)TIM: A Novel Quality of Service Metric for Tactile InternetProceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023)10.1145/3576841.3585917(199-208)Online publication date: 9-May-2023
https://dl.acm.org/doi/10.1145/3576841.3585917
Wei XYao YWang HZhou L(2023)Perception-Aware Cross-Modal Signal Reconstruction: From Audio-Haptic to VisualIEEE Transactions on Multimedia10.1109/TMM.2022.319430925(5527-5538)Online publication date: 2023
https://doi.org/10.1109/TMM.2022.3194309
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents