JP4364024B2 - Mobile device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a portable terminal by which a user can simultaneously and comfortably listen to reproduced sound and incoming tone even if there is an incoming call when sound reproduction is already executed. <P>SOLUTION: If a communication part 101 receives a request for the sound reproduction of incoming call notification when music data stored in a storage part 102 are read out and the music is reproduced from a sound output part 105 by the music data, a control part 107 controls a 3D sound preparing part 106 and forms the music data, by which the music data can be obtained from a sound source by setting a sound source position at an arbitrary position in a three-dimensional space, to the music data during listening. The control part 107 also forms incoming tone data, by which the incoming tone can be obtained from a sound source by setting the sound source at another arbitrary position in the three-dimensional space, to the incoming tone. The formed music data and incoming tone data are supplied to a sound output part 105. By this, the user can hear the music and the incoming tone as if they are obtained from the sound sources at different positions. <P>COPYRIGHT: (C)2005,JPO&amp;NCIPI

Description

  The present invention relates to a portable terminal such as a cellular phone, and more particularly to a portable terminal that simultaneously reproduces a notification sound such as an incoming call or mail reception when sound reproduction is already in progress.

Conventionally, when a plurality of sounds need to be played back, if there is an incoming call while playing music data, the process of fading out the ringtone while outputting the music data being played back, There has been proposed a mobile phone having a function of reproducing music being viewed as BGM even during a call (see, for example, Patent Document 1).
Japanese Patent Laid-Open No. 2002-199056

  In a mobile phone, if there is an incoming call while a sound is already being reproduced for music viewing or the like, it may be necessary to reproduce the incoming call notification sound.

  In the conventional portable terminal described in Patent Document 1, when an incoming call is received during music playback, the ringing tone is faded in and output while fading out the music being played back, and the tone is gradually shifted to the ring tone. In addition, even during a call, the music being played can be used as a BGM during a call by mixing the music data with the call sound.

  However, in such a conventional technique, when an incoming call is received, it fades out during the music reproduction, and only the ring tone is reproduced. Even if the user listens to music, it becomes impossible for the user to listen to the incoming call because the user's conventional viewing environment is broken, which is not comfortable for the user. In view of this, a method of synthesizing and reproducing the sound being played and the ringtone can be considered. However, since these sounds are mixed, there is a problem that it is difficult to distinguish only the sound data to be heard, and it feels uncomfortable.

The object of the present invention is to comfortably view the sound data being played back and the sound data required to be played back simultaneously, even if there is a sound playback request such as ringtone data when the sound data has already been played back. An object of the present invention is to provide a portable terminal that can be used .

A portable terminal according to the present invention includes a storage unit that stores sound data, an input unit that performs input by external voice input or operation, a communication unit that performs communication with the outside, and a sound output that outputs sound data An image display unit that displays character data and image data, a 3D sound creation unit that sets the position of a sound image for localization of the sound image in a three-dimensional space, and creates 3D (three-dimensional) sound data ; A control unit that controls the storage unit, the input unit, the communication unit, the sound output unit, the image display unit, and the 3D sound generation unit, and the control unit outputs sound data by the sound output unit when there is a request for sound reproduction. Is generated, 3D sound data is created for the sound data for which sound reproduction is required, the sound data being reproduced and the created 3D sound data are mixed, and the mixed sound data is output to the sound output unit. Control the 3D sound generator so that However, when sound data is not being output by the sound output unit when there is a request for sound reproduction, 3D sound data is not created for the sound data for which sound reproduction is requested, and the sound data for which sound reproduction is requested is generated. The 3D sound creation unit is controlled so as to be input to the sound output unit .

According to the present invention, even when there is a request for sound reproduction such as ringtone data reproduction when sound data has already been reproduced, the sound data being reproduced and the sound data for which sound reproduction is requested are simultaneously displayed. A portable terminal that can be comfortably viewed can be provided.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a configuration diagram showing an embodiment of a portable terminal according to the present invention, in which 100 is a portable terminal of this embodiment, 101 is a communication unit, 102 is a storage unit, 103 is an input unit, 104 is an image display unit, Reference numeral 105 denotes a sound output unit, 106 denotes a 3D sound creation unit, 107 denotes a control unit, and 108 denotes an antenna.

  In the figure, a portable terminal 100 holds a communication unit 101 that communicates with another terminal or an external device (not shown) of the portable terminal 100 via an antenna 108 and sound data such as voice and music. Storage unit 102, input unit 103 for inputting sound and various operations, image display unit 104 for displaying character data and image data, 3D sound generating unit 106 for generating 3D (three-dimensional) sound by localizing a sound image, sound A sound output unit 105 that outputs data, a control unit 107 that controls each of the above units, and the like are included.

  The 3D sound forming unit 106 localizes (determines the position of the sound image) a sound image (a spatial image of the sound perceived with respect to the sound source) at an arbitrary position in the three-dimensional (3D) space with respect to the sound data. From the input monaural sound data, a sound that is generated from a sound source set by sound image localization and enters the user's left ear and a sound that enters the right ear is generated. As described above, a technique for creating a 3D (dimensional) sound in which a sound image is localized at an arbitrary position in a three-dimensional space and a sound can be heard from an arbitrary place is common. The method described in JP-A-126898 can be used. This will be described below.

  Delay coefficient for each predetermined sound image localization direction (sound source direction set corresponding to sound data viewed counterclockwise with the user's front in front being 0 ° and right behind is 180 °) (for example, every 10 °) , Filter coefficients and amplification coefficients are set (note that filter coefficients and amplification coefficients are also set for each of the left and right ears). First, by using the delay coefficient, the input monaural sound From the data, the first sound data for the left ear and the second sound data for the right ear, which have different time differences according to the delay coefficient (that is, according to the sound image localization direction), are generated. When the sound image localization direction is in the range of 0 ° to 180 °, the second sound data is delayed with respect to the first sound data, and the time difference between these is minimum when the sound image localization direction is 0 ° and maximum when it is 90 °. , 180 ° is the minimum. On the other hand, when the sound image localization direction is in the range of 180 ° to 360 °, the first sound data is delayed with respect to the second sound data, and the time difference between them is minimum when the sound image localization direction is 180 °, 270 Maximum at 360 °, minimum at 360 ° (= 0 °).

  Next, by using the left ear filter coefficient as the coefficient of the sixth-order FIR filter (finite length impulse response type filter) for the first sound data for the left ear, the impulse sequence of the head acoustic transfer function is obtained. Perform the convolution process. The same applies to the second sound data for the right ear, and the impulse of the head acoustic transfer function is obtained by using the filter coefficient for the right ear as a coefficient of a sixth-order FIR filter (finite length impulse response type filter). Perform the process of folding the column. Here, the head-related acoustic transfer function is the transfer function from the sound source to the ear drum, and the sound waves from the sound source are affected by reflection, diffraction, and resonance of the head, pinna, and shoulder. It is a function representing the transmission system up to the ear. A sound image is localized by processing using such a function.

  The first and second sound data processed by the head acoustic transfer function in this manner are amplified using the amplification factors for the left ear and the right ear, and the first and second sound levels change depending on the direction of the sound source. Second sound data is obtained. For example, when the direction of the sound source is 90 °, the sound pressure level of the sound by the first sound data entering the left ear is the maximum, and the sound pressure level of the sound by the second sound data entering the right ear is the minimum. It becomes.

  Through the above processing by the 3D sound forming unit 106, the sound image is localized in the sound image localization direction corresponding to the delay coefficient, the head acoustic transfer function, and the amplification coefficient to be used, and the sound data for the left ear output from this sound image, Sound data for the ear is generated. Such sound data is supplied to the sound output unit 105, and a user who listens to the sound reproduced by the sound output unit 105 sounds as if the sound was emitted from a localized sound image (sound source).

  The control unit 107 determines where the sound image is localized in the three-dimensional space, whereby the 3D sound forming unit 106 sets the sound source at the position determined by the control unit 107.

  The sound output unit 105 is built in the housing of the portable terminal 100 and is configured by two speakers that reproduce 3D sound, but these speakers may be arranged outside the housing. Further, the speaker may have any configuration as long as 3D sound reproduction is possible. Furthermore, instead of the speaker, it may be configured with headphones or the like, or may be configured with a playback device outside the terminal capable of 3D sound playback. The storage unit 102 is normally configured inside the housing, but may be a removable memory card or a storage device outside the housing. The input unit 103 is provided with operation keys and operation buttons, but may further be used for sound input and image input in combination with a camera, microphone, or the like provided inside or outside the housing. .

  FIG. 2 is a flowchart showing a specific example of the processing procedure of this embodiment.

  In the figure, when there is an instruction to start processing, the processing is started (S (step) 201), and the control unit 107 first determines whether there is a sound reproduction request event (S202). This sound reproduction request event is an event in which there is an incoming call, mail, WEB, or the like in the communication unit 101 of the mobile terminal 100, and the reproduction of the ringtone for incoming notification is requested. If there is no sound reproduction request event for the incoming call notification, the process ends (S209).

  If there is a sound reproduction request event for an incoming call notification, the incoming sound data for notifying the incoming call is extracted from the storage unit 102 as the sound data corresponding to the request event (S203).

  Next, the control unit 107 determines whether or not there is currently playing music (S204), and when there is no music data currently being played and only the ring tone needs to be played, Then, the setting is made not to localize the ringtone and the ringtone data is transferred to the 3D sound creation unit 106. The 3D sound creation unit 106 does not localize the received ringtone data according to the setting instruction, and outputs the received ringtone data as it is to the sound output unit 105 as it is (S205) and reproduces it (S208). Thus, the process ends (S209).

  If there is music data currently being played back, the control unit 107 determines the position of the sound source for sound image localization in the three-dimensional space for the ring tone corresponding to the requested event, with respect to the 3D sound creation unit 106. This is notified and ringtone data is transferred. The 3D sound creation unit 106 sets a sound source position for the ring tone data based on the notification from the control unit 107, and creates a 3D sound so that the received ring tone data can be heard from the set sound source position (S206). ) The ring tone data that has been localized and the music data that is already being reproduced are mixed to form a reproduced sound (S207). The formed reproduced sound is supplied to the sound output unit 105 (S208), and the process ends (S209). As a result, the sound output unit 105 sounds as if the ringtone and music were obtained from sound sources having different positions.

  Note that when there is no music data currently being played back and only the ring tone needs to be played, the sound image localization of the ring tone is not performed. However, in this case, the sound image localization may be performed. The 3D sound creation unit 106 enables sound image localization that can be heard from an arbitrary position in a three-dimensional space, and of course, may be only left and right sound image localization as in a conventional stereo speaker.

  The image display unit 104 creates and displays a positional relationship diagram showing the sound source position in which the individual sound data of the created 3D sound is set. The positional relationship diagram to be created may be a map such as a map showing the positional relationship of sound sources in a two-dimensional space, or may be a three-dimensional image such as CG (Computer Graphics). In addition to the sound source position, the illustrated contents can describe information of sound data such as volume, sound direction, and spread, or can be illustrated by marks or shapes.

  FIG. 3 is a diagram illustrating a state in which the user is listening to the 3D sound.

  In the figure, a user 301 can listen to the reproduced sound generated as described above by the sound output unit 105 (FIG. 1) of the mobile terminal 100. For example, when there is a ring tone during music playback, different sounds are heard from different directions so that the music 302 being played is heard from the left rear of the user 301 and the ring tone 303 is heard from the front of the user's right hand. hear.

  At this time, in the image display unit 104, a positional relationship diagram 401 showing the sound source positions of the respective sound data is displayed as shown in FIG. In FIG. 4, reference numeral 402 denotes a music sound source position, and reference numeral 403 denotes a ring tone sound source position.

  When the user performs a predetermined operation on the positional relationship diagram 401 using the input unit 103, the settings of the sound source and sound data in the positional relationship diagram 401 can be changed. For example, by operating the input unit 103, the illustrated sound source is selected, erased, and moved (for example, a sound source to be moved is specified by a cursor and determined, and then the cursor is moved to a new setting position. The sound source at the sound source positions 402 and 403 is represented by a mark having a different shape according to the function at that time, and the mark is moved or its shape is changed (for example, a cursor). By specifying the mark (sound source) to be changed with and selecting the desired shape mark from the menu using the operation keys and buttons, the corresponding playback sound can be eliminated, the sound source position can be changed, It is possible to change the volume, direction, and sound spread.

  FIG. 5 is a diagram showing another specific example of the positional relationship diagram.

  FIG. 5A shows that the sound source 503 is set by the user so that the alarm sound for time notification can be heard from the front of the right hand of the user 301, and the sound source 502 is set so that music can be heard from the entire rear side of the user 301. The positional relationship diagram 402 in the case of being performed is shown. FIG. 5B shows that the sound source 603 is set so that the mail reception sound can be heard from the front side of the user 301 and the sound source 602 is set so that the ring tone can be heard from the front side of the user 301. The positional relationship section 403 is shown when the sound source 604 is set so that the sound of the radio broadcast can be heard from behind the left hand.

  In this way, it is possible to set a sound source of arbitrary sound data at an arbitrary position. For example, the sound data of the set event may not exist alone as sound data included in a moving image or sound data in content, or may be 3D sound. As sound data, sound data included in music or video for viewing, ringtone for receiving mail, alarm sound such as time notification and battery level notification, operation sound of mobile terminal 100 by user, call The voice of the other party, the voice of the other party in the chat, the sound effect when browsing the web, the music attached to the received mail, the user's voice and external sound from the microphone, the music from an external device such as an audio device, from the camera Sound data included in the captured image or video, sound data such as television broadcast or radio broadcast from the receiver may be used.

  In addition, here, the process for playing music for viewing and the ring tone for notifying the incoming call has been described. However, the process can also be applied to any sound playback other than music playback and ring tone playback. The processing for the sound reproduction request of the above sound data can be applied.

  In this embodiment, the control unit 107 receives the sound reproduction request event from the communication unit 101. However, for example, the sound reproduction request event is received from a device other than the communication unit 101, such as the storage unit 102 or the input unit 103. You may make it receive. The sound reproduction request event includes an event due to an incoming call from the communication unit 101 or an email, an event due to a user operation from the input unit 103, a battery remaining amount notification to the control unit 107, a time notification event, or the like. In this embodiment, if there is sound data that is already being reproduced when there is a sound reproduction request event, the sound data corresponding to the sound reproduction request event is localized and mixed with the sound data that is already being reproduced. However, the sound data that is already being played may be localized so that it can be heard from any sound source position, and the sound data may be mixed with the sound data corresponding to the sound request event that has been localized. May be. For example, when playing a ring tone during music playback, the music being played may be heard from the back of the user, and the newly played ring tone may be played with the sound image localized so that it can be heard from the user. The sound data for sound image localization may be three or more sounds. For example, when there is a playback request event for a time notification alarm sound while music is being played from the back of the user and a ring tone is being played from the front of the user, the music can be heard from the back of the user. The ringtones that have been heard from the front of the user can be heard from the right direction of the user, and the sound images can be localized, mixed, and reproduced so that the alarm sound can be heard from the front of the user.

  Furthermore, in this embodiment, the processing in the case where a new sound needs to be reproduced has been described. However, when sound data that does not need to be reproduced occurs in the sound data being reproduced, this sound is reproduced. The sound source of the data is removed, and at the same time, the remaining sound data being reproduced may be re-localized and corrected so that it can be heard from a new sound source position. For example, if the incoming call is interrupted and it is no longer necessary to play the ringtone while playing the sound data so that the ringtone can be heard from the front and the music can be heard from the back, the music data can be repositioned and the music data You may make it reproduce | regenerate so that only can be heard from the front.

  When sound data that is already being played is localized and moved to a different sound source position, the original sound source is moved so that the sound source gradually moves from the existing sound source position to the newly set sound source position. A position between the position and a newly set sound source position is calculated, and a 3D sound is generated such that the sound source position moves step by step using the calculated sound source position. If the sound source is suddenly changed while the user is listening to the sound data, the user may lose sight of the sound data. As described above, the sound source position of the sound data is gradually moved, so that the user can naturally follow the sound data and hear it. The position of the moving process may be calculated on a straight line connecting the original sound source position and the newly set sound source position, or may be calculated on a circumference centered on the user.

  In addition, when it is necessary to play both sound data and image data, such as video data and sound data such as TV images, or both ringtone and image are played when an incoming call is received. The image data is displayed on the image display unit 104. In this case, the control unit 107 determines whether there is a sound reproduction request event. If there is a sound reproduction request event, the control unit 107 obtains sound data corresponding to the sound reproduction request event, and image data attached to the sound data is obtained. It is determined whether or not it exists, and if it exists, this image data is obtained and displayed on the image display unit 104.

  The image display unit 104 also selects image data to be displayed from among the displayable image data such as the positional relationship diagram of the sound source of the created sound data and the image received from the control unit 107, and displays this. At that time, image data to be displayed is selected from the displayable image data in accordance with the number of image data to be displayed and the sound source positional relationship of sound data related to the image data to be displayed.

  For example, when there is no image data received from the control unit 107 with respect to one or more sound data, a positional relationship diagram of the sound source of the sound data is displayed. If there is only one image data received from the control unit 107, the image data is always displayed. In addition, when there are a plurality of displayable image data (that is, when there is image data for at least two or more sound data from a plurality of sound sources), the sound source positions of these sound data The image data to be displayed is selected based on the sound volume, direction, direction, and spread. For example, among a plurality of displayable image data, image data for sound data whose sound source position is closer to the user, image data for sound data whose sound source position is close to the user's front, or image data for sound data having a high volume Alternatively, image data corresponding to sound data that meets other conditions is displayed. If there is no such image data, only the positional relationship diagram of the sound source is displayed.

  As described above, when there are a plurality of displayable image data, the image display unit 104 displays all the image data at the same time or synthesizes the image data related in the positional relationship diagram of the sound source of the sound data. It is also possible to display it.

  As described above, in this embodiment, when a plurality of sounds need to be reproduced, each sound that needs to be reproduced is generated and reproduced as a 3D sound generated from different sound source positions. Therefore, the user can obtain the sound information comfortably and naturally.

  In addition, even when listening to music through headphones and not being able to hear surrounding sounds, the external environmental sound input from the microphone is sounded and reproduced in the back of the user, so the user is looking backward. Can listen to music and listen to the noises of visitors and infants and traffic.

  Also, since different sound data is reproduced from a plurality of directions, a natural environment can be obtained without changing from an environment where telephones and AV devices are reproduced at a time in a normal room. Therefore, since it is possible to listen carefully to one direction, it is easy to distinguish a specific sound, such as listening carefully only with respect to a certain sound from among a plurality of sounds. If you want to listen with extreme caution, the sound source is localized near the ear, and the sound data that you just listen to while listening to BGM, etc., is localized later or far away, depending on the importance of the sound. It is also possible to customize the position. Also, according to this method, the setting of the sound source position is intuitive and convenient because it is only necessary to imagine a normal room.

Further, by displaying the positional relationship diagram of the sound source, the user can more easily grasp the sound source of the 3D sound and can easily hear it. Moreover, the sound source position setting of the sound data can be performed intuitively using a positional relationship diagram. Further, as described above, since the display of the related image data is automatically determined according to the sound source of the sound data, it is not necessary for the user to make detailed settings and is easy to use.

  When movement of the sound source occurs, if the sound source suddenly changes, the user may lose sight of the sound that has been heard so far, and it may be difficult to view the sound. However, in this embodiment, since the sound source can be gradually moved as described above, the user can naturally follow the same sound and can continue viewing without losing sight.

  As described above, in this embodiment, the sound that is already being played and the notification sound corresponding to an event such as an incoming call or mail reception can be heard at the same time by localizing the sound image at an arbitrary position, without destroying the natural viewing environment. Therefore, it is possible to provide a mobile terminal that is easy to use and can be distinguished without a sense of incongruity.

It is a block diagram which shows one Embodiment of the portable terminal by this invention. It is a flowchart which shows a specific example of the process sequence by embodiment shown in FIG. It is the figure which imaged the use condition of embodiment shown in FIG. It is a figure which shows one specific example of the positional relationship figure of the sound source displayed on the image display part in FIG. 1 with respect to the use condition shown in FIG. It is a figure which shows the other specific example of the positional relationship figure of the sound source set in embodiment shown in FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Portable terminal 101 Communication part 102 Memory | storage part 103 Input part 104 Image display part 105 Sound output part 106 3D sound preparation part 107 Control part 108 Antenna 301 User 302 Music 303 Ringtone 401-403 Positional relationship diagram of sound source 402 Music sound source 403 Sound source of ring tone 502 Sound source of music 503 Sound source of alarm sound 602 Sound source of ring tone 603 Sound source of mail reception sound 604 Sound source of radio broadcast sound

Claims (6)

A storage unit for storing sound data;
An input unit for inputting by voice input or operation from the outside,
A communication unit that communicates with the outside;
A sound output unit for outputting sound data ;
An image display unit for displaying character data and image data;
A 3D sound creating unit for creating a 3D (3D) sound data by setting a position of the sound image for localization of the sound image in a 3D space ;
A control unit for controlling the storage unit, the input unit, the communication unit, the sound output unit, the image display unit, and the 3D sound creation unit;
With
The control unit
When sound data is being output by the sound output unit when there is a request for sound reproduction, 3D sound data is created for the sound data for which sound reproduction is requested, and the sound data being reproduced and the creation Controlling the 3D sound generation unit to mix the 3D sound data and the mixed sound data to be input to the sound output unit,
If sound data is not being output by the sound output unit when there is a request for sound reproduction, 3D sound data is not created for the sound data for which sound reproduction is requested, and the sound data for which sound reproduction is requested The mobile terminal is characterized in that the 3D sound creating unit is controlled so as to be input to the sound output unit . In claim 1,
The mobile terminal characterized in that the sound data for which sound reproduction is required is ring tone data for notifying an incoming call . The portable terminal according to claim 1 or 2,
The 3D sound creation unit creates a positional relationship diagram showing a positional relationship between the setting position of the sound source and the user,
The mobile terminal, wherein the image display unit displays the positional relationship diagram. The mobile terminal according to claim 3,
An operation on the positional relationship diagram displayed on the image display unit is possible on the input unit,
The mobile terminal having a function of enabling the setting position of the sound source to be changed in response to the operation at the input unit under the control of the control unit. In the portable terminal as described in any one of Claims 1-4,
The 3D sound creation unit is moved and changed according to a request to move and change the setting position of the sound source from the input means under the control of the control unit. One or more intermediate positions are sequentially calculated between the sound source and the new set position of the sound source, and each time the intermediate position is calculated, the sound data output from the intermediate position is set as the sound source position. Has the ability to create,
A portable terminal characterized by gradually moving a sound source from a set position of a sound source before movement or change to a new set position of a specified sound source after change. In the portable terminal as described in any one of Claims 1-5,
The mobile terminal characterized in that the image display unit has a function of displaying image data corresponding to a sound source whose position is set.

Images