More Web Proxy on the site http://driver.im/

research-article

Perceived Synchronization of Mulsemedia Services

Authors:

Gabriel-Miro Muntean,

Gheorghita GhineaAuthors Info & Claims

IEEE Transactions on Multimedia, Volume 17, Issue 7

Pages 957 - 966

https://doi.org/10.1109/TMM.2015.2431915

Published: 01 July 2015 Publication History

Abstract

Multimedia synchronization involves a temporal relationship between audio and visual media components. The presentation of “in-sync” data streams is essential to achieve a natural impression, as “out-of-sync” effects are often associated with user quality of experience (QoE) decrease . Recently, multi-sensory media (mulsemedia) has been demonstrated to provide a highly immersive experience for its users. Unlike traditional multimedia, mulsemedia consists of other media types (i.e., haptic, olfaction, taste, etc.) in addition to audio and visual content. Therefore, the goal of achieving high quality mulsemedia transmission is to present no or little synchronization errors between the multiple media components. In order to achieve this ideal synchronization, there is a need for comprehensive knowledge of the synchronization requirements at the user interface. This paper presents the results of a subjective study carried out to explore the temporal boundaries within which haptic and air-flow media objects can be successfully synchronized with video media. Results show that skews between sensorial media and multimedia might still give the effect that the mulsemedia sequence is “in-sync” and provide certain constraints under which synchronization errors might be tolerated. The outcomes of the paper are used to provide recommendations for mulsemedia service providers in order for their services to be associated with acceptable user experience levels, e.g. haptic media could be presented with a delay of up to 1 s behind video content, while air-flow media could be released either 5 s ahead of or 3 s behind video content.

References

[1]

Z. Yuan, G. Ghinea, and G.-M. Muntean, “Beyond multimedia adaptation: Quality of experience-aware multi-sensorial media delivery,” IEEE Trans. Multimedia, vol. 17, no. 1, pp. 104–117, Jan. 2015.

[2]

Z. Yuan, S. Chen, G. Ghinea, and G.-M. Muntean, “User quality of experience of mulsemedia applications,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 11, no. 1s, pp. 15:1–15:19, Sep. 2014.

[3]

Z. Yuan, G. Ghinea, and G.-M. Muntean, “Quality of experience study for multiple sensorial media delivery,” in Proc. IEEE Int. Wireless Commun. Mobile Comput. Conf., Nicosia, Cyprus, Aug. 2014, pp. 1142–1146.

[4]

O. Ademoye and G. Ghinea, “Synchronization of olfaction-enhanced multimedia,” IEEE Trans. Multimedia, vol. 11, no. 3, pp. 561–565, Apr. 2009.

Digital Library

[5]

G. Ghinea and O. Ademoye, “User perception of media content association in olfaction-enhanced multimedia,” ACM Trans. Multimedia Comput., Commun. Appl., vol. 8, no. 4, pp. 52:1–52:19, Nov. 2012.

[6]

J. G. Apostolopoulos, P. A. Chou, B. Culbertson, T. Kalker, M. D. Trott, and S. Wee, “The road to immersive communication,” Proc. IEEE, vol. 100, no. 4, pp. 974–990, Apr. 2012.

[7]

C. Timmerer, M. Waltl, B. Rainer, and N. Murray Sensory experience: Quality of experience beyond audio-visual Quality of Experience: Advanced Concepts, Applcations and Methods S. Moller and A. Raake, Eds., Berlin, Germany: Springer-Verlag, 2014, pp. 351–365.

[8]

S. Ud Din and D. Bulterman, “Synchronization techniques in distributed multimedia presentation,” in Proc. 4th Int. Conf. Adv. Multimedia, Apr. 2012, pp. 1–9.

[9]

F. Boronat, M. Montagud, H. M. Stokking, and O. Niamut, “The need for inter-destination synchronization for emerging social interactive multimedia applications,” IEEE Commun. Mag., vol. 50, no. 11, pp. 150–158, Nov. 2012.

[10]

R. Steinmetz, “Human perception of jitter and media synchronization,” IEEE J. Sel. Areas Commun., vol. 14, no. 1, pp. 61–72, Jan. 1996.

Digital Library

[11]

G. Blakowski and R. Steinmetz, “A media synchronization survey: Reference model, specification, and case studies,” IEEE J. Sel. Areas Commun., vol. 14, no. 1, pp. 5–35, Jan. 1996.

Digital Library

[12]

D. A. Washburn, L. M. Jones, R. V. Satya, C. A. Bowers, and A. Cortes, “Olfactory use in virtual environment training,” Model. Simul. Mag., vol. 2, no. 3, pp. 19–25, 2003.

[13]

Z. Huang, K. Nahrstedt, and R. Steinmetz, “Evolution of temporal multimedia synchronization principles: A historical viewpoint,” ACM Trans. Multimedia Comput., Commun. Appl., vol. 9, no. 1s, pp. 34:1–34:23, Oct. 2013.

[14]

M. Montagud, F. Boronat, and H. Stokking, “Early event-driven (EED) RTCP feedback for rapid IDMS,” in Proc. 21st ACM Int. Conf. Multimedia, New York, NY USA, 2013, pp. 323–332.

[15]

C. Concolato, S. Thomas, R. Bouqueau, and J. Le Feuvre, “Synchronized delivery of multimedia content over uncoordinated broadcast broadband networks,” in Proc. ACM Multimedia Syst. Conf., New York, NY USA, 2012, pp. 227–232.

[16]

R. Mekuria, P. Cesar, and D. Bulterman, “Digital TV: The effect of delay when watching football,” in Proc. ACM Eur. Conf. Interactive TV Video, New York, NY USA, 2012, pp. 71–74.

[17]

N. Murray, Y. Qiao, B. Lee, and G.-M. Muntean, “User profile based perceived olfactory and visual media synchronization,” ACM Trans. Multimedia Comput. Commun. Appl., vol. 10, no. 1, pp. 11:1–11:24, Jan. 2014.

[18]

N. Murray, B. Lee, Y. Qiao, and G.-M. Muntean, “Multiple-scent enhanced multimedia synchronization,” ACM Trans. Multimedia Comput. Commun. Appl. (TOMM), vol. 11, no. 1s, pp. 12:1–12:28, Sep. 2014.

[19]

N. Murray, Y. Qiao, B. Lee, A. K. Karunakar, and G.-M. Muntean, “Subjective evaluation of olfactory and visual media synchronization,” in Proc. ACM Multimedia Syst. Conf., Oslo, Norway, Feb. 2013, pp. 162–171.

[20]

A. Bodnar and R. Corbett, “AROMA: Ambient awareness through olfaction in a messaging application—Does olfactory notification make ‘scents’?,” in Proc. ACM Int. Conf. Multimodal Interfaces, Oct. 2004, pp. 183–190.

[21]

H. Q. Dinh, N. Walker, L. F. Hodges, C. Song, and A. Kobayashi, “Evaluating the importance of multi-sensory input on memory and the sense of presence in virtual environments,” in Proc. Virtual Reality Annu. Int. Symp., 1999, pp. 222–228.

[22]

Subjective Video Quality Assessment Methods for Multimedia Applications, ITU-T Rec. P.910 Int. Telecommun. Union, Geneva, Switzerland, Sep. 1999.

[23]

Subjective Audiovisual Quality Assessment Methods for Multimedia Applications, ITU-T Rec. P.911 Int. Telecommun. Union, Geneva, Switzerland, Dec. 1998.

[24]

Methods for the Subjective Assessment of Video Quality, Audio Quality and Audiovisual Quality of Internet Video and Distribution Quality Television in Any Environment, ITU-T Rec. P.913 Int. Telecommun. Union, Geneva, Switzerland, Jan. 2014.

[25]

L. Cronbach, “Coefficient alpha and the internal structure of tests,” Psychometrika, vol. 16, no. 3, pp. 297–334, 1951.

[26]

D. Streiner, “Starting at the beginning: An introduction to coefficient alpha and internal consistency,” J. Personality Assessment, vol. 80, no. 1, pp. 99–103, 2003.

[27]

D. W. Nordstokke and B. D. Zumbo, “A cautionary tale about Levene's tests for equality of variances,” J. Educ. Res. Policy Studies, vol. 7, no. 1, pp. 1–14, 2007.

[28]

D. W. Nordstokke, B. D. Zumbo, S. L. Cairns, and D. H. Saklofske, “The operating characteristics of the nonparametric Levene test for equal variances with assessment and evaluation data,” Practical Assess., Res. Eval., vol. 16, no. 5, p. 2, 2011.

[29]

D. J. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures, 5th ed., London, U.K.: Chapman & Hall, 2011.

[30]

Y. Ishibashi, T. Kanbara, and S. Tasaka, “Inter-stream synchronization between haptic media and voice in collaborative virtual environments,” in Proc. 12th Annu. ACM Int. Conf. Multimedia, New York, NY USA, Oct. 2004, pp. 604–611.

[31]

G. Ghinea and O. A. Ademoye, “Perceived synchronization of olfactory multimedia,” IEEE Trans. Syst., Man Cybern. A, Syst. Humans, vol. 40, no. 4, pp. 657–663, Jul. 2010.

Digital Library

[32]

Q. Zeng, Y. Ishibashi, N. Fukushima, S. Sugawara, and K. E. Psannis, “Influences of inter-stream synchronization errors among haptic media, sound, and video on quality of experience in networked ensenmble,” in Proc. 2nd IEEE Global Conf. Consum. Electron., Oct. 2013, pp. 466–470.

[33]

P. Huang, Q. Zeng, and Y. Ishibashi, “QoE assessment of will transmission using haptics: Influence of network delay,” in Proc. IEEE Global Conf. Consum. Electron., Oct. 2013, pp. 456–460.

[34]

A. Tatematsu, Y. Ishibashi, N. Fukushima, and S. Sugawara, “QoE assessment in haptic media, sound and video transmission: Influences of network latency,” in Proc. IEEE Int. Workshop Commun. Quality Rel., Jun. 2010, pp. 1–6.

[35]

J. M. Silva, M. Orozco, J. Cha, A. E. Saddik, and E. M. Petriu, “Human perception of haptic-to-video and haptic-to-audio skew in multimedia applications,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 9, no. 2, pp. 9:1–9:16, May 2013.

[36]

H. Kim and J. H. Ryu, “A study on the effect of haptic to video time-delay on teleoperation and a comment for improving the performance,” in Proc. Int. Conf. Control, Automat. Syst., Oct. 2011, pp. 1329–1332.

[37]

S. Kameyama and Y. Ishibashi, “Perception of synchronization errors in haptic and visual communications,” Proc. SPIE Optics East, Multimedia Syst. Appl. IX, vol. 6391, no. 8, pp. 639108.1–639108.8, Oct. 2006.

[38]

E. Isomura, S. Tasaka, and T. Nunome, “QoE enhancement in audiovisual and haptic interactive IP communications by media adaptive intra-stream synchronization,” in Proc. IEEE Int. Tech. Conf. IEEE Region 10, Nov. 2011, pp. 1085–1089.

[39]

T. Nakano, Y. Yoshioka, and Y. Yanagida, “Effects of wind source configuration of wind displays on property of wind direction perception,” in Proc. Int. Conf. Adv. Comput.–Human Interactions, 2014, pp. 365–370.

[40]

E. Steinbach, S. Hirche, M. Ernst, F. Brandi, R. Chaudhari, J. Kammerl, and I. Vittorias, “Haptic communications,” Proc. IEEE, vol. 100, no. 4, pp. 937–956, Apr. 2012.

[41]

H. T. Nagle, R. Gutierrez-Osuna, and S. S. Schiffman, “The how and why of electronic noses,” IEEE Spectr., vol. 35, no. 9, pp. 22–31, Sep. 1998.

Digital Library

[42]

L. A. Rowe and R. Jain, “ACM SIGMM retreat report on future directions in multimedia research,” ACM Trans. Multimedia Comput., Commun., Appl., vol. 1, no. 1, pp. 3–13, Feb. 2005.

Digital Library

[43]

D. G. Laing, “Natural sniffing gives optimum odour perception for humans,” Perception, vol. 12, no. 2, pp. 99–117, 1983.

[44]

Z. Yuan and G.-M. Muntean, “A prioritized adaptive scheme for multimedia services over IEEE 802.11 WLANs,” IEEE Trans. Netw. Service Manag., vol. 10, no. 4, pp. 340–355, Dec. 2013.

[45]

J. Xiao, T. Tillo, C. Lin, Y. Zhang, and Y. Zhao, “A real-time error resilient video streaming scheme exploiting late and early-arrival packets,” IEEE Trans. Broadcast., vol. 59, no. 3, pp. 432–444, Sep. 2013.

[46]

C. C. Carbon and H. Leder, “The repeated evaluation technique (RET). A method to capture dynamic effects of innovativeness and attractiveness,” Appl. Cognitive Psychol., vol. 19, no. 5, pp. 587–601, 2005.

[47]

M. Jakesch, M. Zachhuber, H. Leder, M. Spingler, M., and C. C. Carbon, “Scenario-based touching: On the influence of top-down processes on tactile and visual appreciation,” Res. Eng. Design, vol. 22, pp. 143–152, 2011.

[48]

E. J. Rodriguez, T. Nunome, and S. Tasaka, “Multidimensional QoE assessment of multi-view video and audio (MVV-A) IP transmission: The effects of user interfaces and contents,” in Proc. Int. Conf. Adv. Inf. Netw. Appl. Workshops, 2012, pp. 91–98.

[49]

N. Murray, Y. Qiao, B. Lee, G.-M. Muntean, and A. K. Karunakar, “Age and gender influence on perceived olfactory and visual media synchronization,” in Proc. IEEE Int. Conf. Multimedia Expo, Jul. 2013, pp. 1–6.

[50]

Y. Ishibashi, S. Hoshino, Q. Zeng, N. Fukushima, and S. Sugawara, “QoE assessment of fairness in networked game with olfaction: Influence of time it takes for smell to reach player,” Multimedia Syst. J., vol. 20, no. 5, pp. 621–631, May 2014.

Digital Library

[51]

A. Molnar and C. H. Muntean, “Cost-oriented adaptive multimedia delivery,” IEEE Trans. Broadcast., vol. 59, no. 3, pp. 484–499, Sep. 2013.

[52]

O. Ormond, G.-M. Muntean, and J. Murphy, “Economic model for cost effective network selection strategy in service oriented heterogeneous wireless network environment,” in Proc. IEEE/IFIP Netw. Operations Manag. Symp., Apr. 2006, pp. 1–4.

[53]

A. N. Moldovan, S. Weibelzahl, and C. H. Muntean, “Energy-aware mobile learning: Opportunities and challenges,” IEEE Commun. Surveys Tuts., vol. 16, no. 1, pp. 234–265, 2014.

[54]

M. Kennedy, A. Ksentini, Y. Hadjadj-Aoul, and G.-M. Muntean, “Adaptive energy optimization in multimedia-centric wireless devices: A survey,” IEEE Commun. Surveys Tuts., vol. 15, no. 2, pp. 768–786, 2013.

Cited By

Ivanov MMoreno MMuchaluat-Saade D(2024)Automatic Preparation of Sensory EffectsProceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3647615(100-108)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3647615
Fang YLiu QXu YGuo YZhao T(2023)Virtual reality interaction based on visual attention and kinesthetic informationVirtual Reality10.1007/s10055-023-00801-327:3(2183-2193)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1007/s10055-023-00801-3
Tasaka S(2022)An Empirical Method for Causal Inference of Constructs for QoE in Haptic–Audiovisual CommunicationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/347398618:1(1-24)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3473986
Show More Cited By

Index Terms

Perceived Synchronization of Mulsemedia Services
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
2. Information systems
  1. Information systems applications
    1. Multimedia information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

Perceived synchronization of olfactory multimedia

The concept of synchronization is of fundamental importance in multimedia systems and applications. The focus of this this paper is on olfaction-enhanced multimedia, which concerns itself with associating computer-generated smell with other media. ...
Mulsemedia preparation: a new event type for preparing media object presentation and sensory effect rendering
MMSys '19: Proceedings of the 10th ACM Multimedia Systems Conference

In multimedia applications, a short delay in content delivery may cause synchronization failures among media objects, decreasing the presentation quality and user quality of experience. Moreover, some applications incorporate sensory effects to ...
A user perspective of olfaction-enhanced mulsemedia
MEDES '10: Proceedings of the International Conference on Management of Emergent Digital EcoSystems

The focus of this paper is on olfaction-enhanced multimedia applications, that is, combining computer generated smell with other media to enrich the users' experience and perception of a multimedia application. Within the general context of mulsemedia --...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Multimedia

IEEE Transactions on Multimedia Volume 17, Issue 7

July 2015

199 pages

ISSN:1520-9210

Issue’s Table of Contents

Copyright © 2015.

Publisher

IEEE Press

Publication History

Published: 01 July 2015

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ivanov MMoreno MMuchaluat-Saade D(2024)Automatic Preparation of Sensory EffectsProceedings of the 15th ACM Multimedia Systems Conference10.1145/3625468.3647615(100-108)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3625468.3647615
Fang YLiu QXu YGuo YZhao T(2023)Virtual reality interaction based on visual attention and kinesthetic informationVirtual Reality10.1007/s10055-023-00801-327:3(2183-2193)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1007/s10055-023-00801-3
Tasaka S(2022)An Empirical Method for Causal Inference of Constructs for QoE in Haptic–Audiovisual CommunicationsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/347398618:1(1-24)Online publication date: 27-Jan-2022
https://dl.acm.org/doi/10.1145/3473986
Hussain NGhinea G(2022)Guidelines for evaluating wearables’ quality of experience in a mulsemedia contextMultimedia Tools and Applications10.1007/s11042-022-12766-781:30(43283-43314)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s11042-022-12766-7
Alkasasbeh ASpyridonis FGhinea G(2021)When Scents Help Me Remember My PasswordACM Transactions on Applied Perception10.1145/346988918:3(1-18)Online publication date: 20-Aug-2021
https://dl.acm.org/doi/10.1145/3469889
Bi TLyons RFox GMuntean G(2021)Improving Student Learning Satisfaction by Using an Innovative DASH-Based Multiple Sensorial Media Delivery SolutionIEEE Transactions on Multimedia10.1109/TMM.2020.302566923(3494-3505)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1109/TMM.2020.3025669
Kani-Zabihi EHussain NMesfin GCovaci AGhinea G(2021)On the influence of individual differences in cross-modal Mulsemedia QoEMultimedia Tools and Applications10.1007/s11042-020-09757-x80:2(2377-2394)Online publication date: 1-Jan-2021
https://dl.acm.org/doi/10.1007/s11042-020-09757-x
Jost CPévédic BUzan G(2019)MulseBoxProceedings of the 31st Conference on l'Interaction Homme-Machine10.1145/3366550.3372251(1-13)Online publication date: 10-Dec-2019
https://dl.acm.org/doi/10.1145/3366550.3372251
Saleme ECovaci AMesfin GSantos CGhinea G(2019)Mulsemedia DIY: A Survey of Devices and a Tutorial for Building Your Own Mulsemedia EnvironmentACM Computing Surveys10.1145/331985352:3(1-29)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3319853
Josué MMuchaluat-Saade DMoreno MNeto MNovais RFerraz CViana W(2018)Preparation of Media Object Presentation and Sensory Effect Rendering in Mulsemedia ApplicationsProceedings of the 24th Brazilian Symposium on Multimedia and the Web10.1145/3243082.3243098(45-52)Online publication date: 16-Oct-2018
https://dl.acm.org/doi/10.1145/3243082.3243098
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents