Mechanisms for multimodality: taking fiction to another dimension
Pages 135 - 144
Abstract
We present methods for automatically constructing representations of fiction books in a range of modalities: audibly, graphically and as 3D virtual environments. The correspondence between the sequential ordering of events against the order of events presented in the text is used to correctly resolve the dynamic interactions for each representation. Synthesised audio created from the fiction text is used to calibrate the base time-line against which the other forms of media are correctly aligned. The audio stream is based on speech synthesis using the text of the book, and is enhanced using distinct voices for the different characters in a book. Sound effects are included automatically. The graphical representation represents the text (as subtitles), identifies active characters and provides visual feedback of the content of the story. Dynamic virtual environments conform to the constraints implied by the story, and are used as a source of further visual content. These representations are all aligned to a common time-line, and combined using sequencing facilities to provide a multimodal version of the original text.
References
[1]
Akerberg, O., Svensson, H., Schulz, B., and Nugues, P. 2003. Carsim: An automatic 3D text-to-scene conversion system applied to road accident reports. In Research Notes and Demonstrations Conference Companion, 10th Conference of the European Chapter of the Association of Computational Linguistics, Association for Computational Linguistics, Budapest, Hungary, 191--194.
[2]
Back, M., Gold, R., and Kirsch, D. The SIT book: audio as affective imagery for interactive storybooks. In CHI '99: CHI '99 extended abstracts on Human factors in computing systems, ACM Press, New York, NY, USA, 202--203.
[3]
Badler, N. I., Bindiganavale, R., Allbeck, J., Schuler, W., Zhao, L., and Palmer, M. 2000. Parameterized action representation for virtual human agents. Embodied conversational agents. MIT Press, ch. 9, 256--284.
[4]
Benhamou, F., Goualard, F., Granvilliers, L., and Puget, J.-F. 1999. Revising hull and box consistency. In Proceedings of the sixteenth International Conference on Logic Programming (ICLP'99), MIT Press, Las Cruces, New Mexico, United States, 230--244.
[5]
Benhamou, F., Goualard, F., Languénou, É., and Christie, M. 2004. Interval constraint solving for camera control and motion planning. ACM Transactions on Computational Logic (TOCL) 5, 4, 732--767.
[6]
Billinghurst, M., Kato, H., and Poupyrev, I. 2001. The magicbook - moving seamlessly between reality and virtuality. Computer Graphics and Applications 21(3) (May-June), 2--4.
[7]
Bindiganavale, R., Schuler, W., Allbeck, J. M., Badler, N. I., Joshi, A. K., and Palmer, M. 2000. Dynamically altering agent behaviors using natural language instructions. In AGENTS '00: Proceedings of the fourth international conference on Autonomous agents, ACM Press, New York, NY, USA, 293--300.
[8]
Chu, Y.-C., Witten, I. H., Lobb, R., and Bainbridge, D. 2003. How to turn the page. In JCDL '03: Proceedings of the 3rd ACM/IEEE-CS joint conference on Digital libraries, IEEE Computer Society, Houston, Texas, 186--188.
[9]
Clay, S. R., and Wilhelms, J. 1996. Put: Language-based interactive manipulation of objects. IEEE Computer Graphics and Applications 16, 2 (March), 31--39.
[10]
Coyne, B., and Sproat, R. 2001. Wordseye: an automatic text-to-scene conversion system. In SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM Press, New York, NY, USA, 487--496.
[11]
Fellbaum, C., Ed. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, USA.
[12]
Glass, K., and Bangay, S. 2005. Evaluating parts-of-speech taggers for use in a text-to-scene conversion system. In SAICSIT '05: Proceedings of the 2005 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countries, South African Institute for Computer Scientists and Information Technologists, Republic of South Africa, 20--28.
[13]
Glass, K., and Bangay, S. 2006. Hierarchical rule generalisation for speaker identification in fiction books. In SAICSIT '06: Proceedings of the 2006 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing couuntries, South African Institute for Computer Scientists and Information Technologists, Republic of South Africa, 31--40.
[14]
Glass, K., and Bangay, S. 2007. Constraint-based conversion of fiction text to a time-based graphical representation. In SAICSIT '07: 2007 annual research conference of the South African institute of computer scientists and information technologists, South African Institute for Computer Scientists and Information Technologists, Republic of South Africa.
[15]
Glass, K. R., Morkel, C., and Bangay, S. D. 2006. Duplicating road patterns in South African informal settlements using procedural techniques. In Afrigraph '06: Proceedings of the 4th international conference on Computer graphics, virtual reality, visualisation and interaction in Africa, ACM Press, New York, NY, USA, 161--169.
[16]
Hood, M. 2004. Creating a voice for festival speech synthesis system. Tech. rep., Computer Science Department, Rhodes University, Grahamstown, South Africa, November.
[17]
Joshi, D., Wang, J. Z., and Li, J. 2004. The story picturing engine: finding elite images to illustrate a story using mutual reinforcement. In Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, ACM Press, New York, NY, USA, 119--126.
[18]
Lu, R., and Zhang, S. 2002. Automatic Generation of Computer Animation: using AI for movie animation, vol. 2160 of Lecture Notes in Computer Science. Springer, Berlin.
[19]
Ma, M. E. 2002. CONFUCIUS: An intelligent multimedia storytelling interpretation and presentation system. Tech. rep., School of Computing and Intelligent Systems, University of Ulster, Magee, September.
[20]
Moore, R. E. 1966. Interval Analysis. Prentice-Hall, Inc., New Jersey, USA.
[21]
Morkel, C., and Bangay, S. 2006. Procedural modeling facilities for hierarchical object generation. In Afrigraph '06: Proceedings of the 4th international conference on Computer graphics, virtual reality, visualisation and interaction in Africa, ACM Press, New York, NY, USA, 145--154.
[22]
Musgrave, F. K., Kolb, C. E., and Mace, R. S. 1989. The synthesis and rendering of eroded fractal terrains. In Proceedings of the 16th annual conference on Computer graphics and interactive techniques, ACM Press, 41--50.
[23]
Piesk, J., and Trogemann, G. 1997. Animated interactive fiction: Storytelling by a conversational virtual actor. In VSMM '97: Proceedings of the 1997 International Conference on Virtual Systems and MultiMedia, IEEE Computer Society, Washington, DC, USA, 100.
[24]
Tabordet, F., Pied, F., and Nugues, P. 1999. Scene visualization and animation from texts in a virtual environment. Journal for the integrated study of artificial intelligence, cognitive science and applied epistemology 15, 4, 339--349.
[25]
Tapanainen, P., and Järvinen, T. 1997. A non-projective dependency parser. In Proceedings of the 5th Conference on Applied Natural Language Processing, Morgan Kaufmann Publishers Inc., Washington, DC, USA, Association for Computational Linguistics, 64--71.
[26]
Zeng, X., Mehdi, Q. H., and Gough, N. E. 2003. Shape of the story: Story visualization techniques. Proceedings of the Seventh International Conference on Information Visualization (IV'03), 144--149.
[27]
Zeng, X., Mehdi, Q. H., and Gough, N. E. 2005. From visual semantic parameterization to graphic visualization. In IV '05: Proceedings of the Ninth International Conference on Information Visualisation (IV'05), IEEE Computer Society, Washington, DC, USA, 488--493.
[28]
Zhang, J., Black, A., and Sproat, R. 2003. Identifying speakers in children's stories for speech synthesis. In Proceedings of EUROSPEECH 2003, Institute for Perceptual Artificial Intelligence, Geneva, Switzerland.
Index Terms
- Mechanisms for multimodality: taking fiction to another dimension
Recommendations
Multimodality in VR: A Survey
Virtual reality (VR) is rapidly growing, with the potential to change the way we create and consume content. In VR, users integrate multimodal sensory information they receive to create a unified perception of the virtual world. In this survey, we review ...
Constraint-based conversion of fiction text to a time-based graphical representation
SAICSIT '07: Proceedings of the 2007 annual research conference of the South African institute of computer scientists and information technologists on IT research in developing countriesThis paper presents a method for converting unrestricted fiction text into a time-based graphical form. Key concepts extracted from the text are used to formulate constraints describing the interaction of entities in a scene. The solution of these ...
Multimodal spatial reference in mediated environments: users' preferences and the pragmatics of pointing and talking
CHI EA '06: CHI '06 Extended Abstracts on Human Factors in Computing SystemsThis paper describes the current results and future developments of a project on multimodal spatial reference in mediated environments. The database consists of video-recorded sessions, with 120 participants in three experimental designs, contrasting ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
October 2007
215 pages
ISBN:9781595939067
DOI:10.1145/1294685
- Conference Chair:
- Hannah Slay,
- Editor:
- Stephen N. Spencer,
- Program Chair:
- Shaun Bangay
Copyright © 2007 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 October 2007
Check for updates
Author Tags
Qualifiers
- Article
Conference
AFRIGRAPH07
Sponsor:
AFRIGRAPH07: 5th International Conference on Computer Graphics, Virtual Reality, Visualisation and Interaction in Africa
October 29 - 31, 2007
Grahamstown, South Africa
Acceptance Rates
Overall Acceptance Rate 47 of 90 submissions, 52%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 352Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)1
Reflects downloads up to 12 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in