JP2024034347A - Sound generation notification device and sound generation notification method - Google Patents

Abstract

To provide a sound generation notification device and a sound generation notification method that can appropriately notify sound generation.SOLUTION: A sound generation notification device 10 according to this embodiment includes: a sound acquisition unit 21 for acquiring sound collected; a sound direction identification unit 25 for identifying a direction of arrival of the sound; a sound content recognition unit 23 for recognizing contents of the sound; a display unit 12 for displaying images; and a display control unit 26 for displaying an image of a virtual body generated based on the direction of arrival and the contents of the sound on the display unit 12.SELECTED DRAWING: Figure 1

Description

The present invention relates to a sound generation notification device and a sound generation notification method that visually notify the generation of sound.

In general, people who are hard of hearing or have hearing impairments (hereinafter referred to as hearing-impaired people) have difficulty or may not be able to hear surrounding sounds (such as everyday sounds and notification sounds). For this reason, there is a known technology that localizes the sound source by collecting the sound with a microphone that can estimate the direction of arrival of this type of ambient sound, and then notifies the user by displaying text or blinking on mobile terminals, home appliances, etc. (for example, Patent Document 1) reference). According to this type of technology, it is possible to easily notify a hearing-impaired person that a sound is occurring in the surrounding area.

JP2019-66529A

However, with conventional technology, it is only possible to notify the occurrence of sound from pre-registered sound sources, making it difficult to recognize in detail the direction of sound arrival, the pitch of the voice, the tone (emotion), etc. There was room for improvement in properly notifying the occurrence of sounds.

The present invention has been made in view of the above, and an object of the present invention is to provide a sound generation notification device and a sound generation notification method that can appropriately notify the generation of sound.

In order to solve the above-mentioned problems and achieve the purpose, a sound generation notification device according to the present invention includes: a sound acquisition unit that acquires collected sound; a sound direction identification unit that identifies the direction of arrival of the sound; It includes a voice content recognition unit that recognizes the content of the voice, a display unit that displays an image, and a display control unit that causes the display unit to display an image of the virtual object that is generated based on the direction of arrival and the content of the voice. .

Further, the sound generation notification method according to the present invention includes a sound acquisition step of acquiring collected sound, a sound direction identification step of identifying the direction of arrival of the sound, and a sound content recognition step of recognizing the content of the sound. The method further includes a display control step of displaying, on a display unit, an image of the virtual object generated based on the arrival direction and content of the audio.

According to the present invention, it is possible to appropriately notify the generation of sound.

FIG. 1 is a block diagram showing a configuration example of a sound generation notification device according to this embodiment. FIG. 2 is a flowchart showing the procedure of operation in the three-dimensional figure projection mode. FIG. 3 is a diagram showing an example of a virtual object displayed in a group meeting in which the three-dimensional figure projection mode is executed. FIG. 4 is a flowchart showing the procedure of operation in the danger/warning mode. FIG. 5 is a diagram showing an example of a virtual object displayed in a city area where the danger/warning notification mode is executed.

FIG. 1 is a block diagram showing a configuration example of a sound generation notification device according to this embodiment. The sound generation notification device 10 is, for example, a glasses-type device worn on the user's head, such as AR (Augmented Reality) glasses, smart glasses, or a head-mounted display. The sound generation notification device 10 visually notifies a hearing-impaired user that a sound has occurred around the user, for example. As shown in FIG. 1, the sound generation notification device 10 includes a sound collection section 11, a display section 12, a communication section 13, and a notification control device 20.

The sound collection unit 11 is a microphone array, and has a plurality of microphones arranged at different positions. The sound collection unit 11 collects sounds that arrive at the sound generation notification device 10, and generates multiple channels of acoustic signals from these collected sounds. The sound collection unit 11 outputs the generated multi-channel audio signals to the audio acquisition unit 21 of the notification control device 20 . Further, as the sound collection unit 11, a microphone capable of estimating the arrival direction of sound, such as a binaural microphone or a stereophonic microphone, may be used.

The display unit 12 is arranged in front of the eyes of the user wearing the sound generation notification device 10 and displays an image, and in this embodiment is, for example, a glass monitor. The glass monitor has a structure that not only displays an image but also allows the field of view to be seen through the glass monitor. On the display unit 12, the display control unit 26 of the notification control device 20 displays an image superimposed on the scenery of the actual field of view.

The communication section 13 is a communication unit. The communication unit 13 may communicate using any method such as the Internet or a mobile phone line. The communication unit 13 performs various communications with, for example, a server device (not shown) based on a control signal output from the communication control unit 27 of the notification control device 20.

The notification control device 20 includes a voice acquisition section 21, a sound source separation section 22, a voice content recognition section 23, an emotion analysis section 24, a voice direction identification section 25, a display control section 26, a communication control section 27, It includes a hearing level determination section 28 and a storage section 29.

The audio acquisition unit 21 acquires multiple channels of audio signals output from the sound collection unit 11 and outputs the acquired audio signals to the sound source separation unit 22 . The sound source separation unit 22 separates the plurality of acoustic signals included in the plurality of channels acquired by the audio acquisition unit 21 into acoustic signals for each sound source. The sound source separation unit 22 may, for example, separate a plurality of acoustic signals in which sounds uttered by multiple people at the same time are mixed into acoustic signals of the voices of each person, or, for example, separate the sound of a car that can be heard outdoors and the utterances of a person. The system separates a plurality of audio signals mixed with the sound of a car into an audio signal of a car running sound and an audio signal of a human voice.

The speech content recognition unit 23 converts the acoustic signal of a human voice into text data by performing speech recognition processing on each acoustic signal whose sound source has been separated. In this case, for sounds other than those uttered by people (for example, car running sounds, alarm sounds, bicycle bells and brake sounds, various alarm sounds, rain sounds, wind sounds, thunder sounds, etc.), which type of sound is used? Determine and recognize the content. An existing method can be used for speech recognition.

The emotion analysis unit 24 analyzes the emotion of the speaker. For example, the emotion analysis unit 24 determines whether the speaker's emotion is positive (calm, happy, etc.) or negative (angry, sad, fear, etc.) based on the volume and tone of the voice in the acoustic signal of the uttered voice. ). The emotion analysis unit 24 may add the recognized voice content to the audio signal of the voice and analyze the emotion of the speaker. Existing methods (eg, OpenSMILE software) can be used to analyze emotions.

The audio direction identification unit 25 identifies the direction of audio coming toward the sound generation notification device 10 . For example, the audio direction identification unit 25 identifies from which direction the collected audio has arrived based on the difference in arrival time of each of the collected audio signals of the plurality of channels. Other existing methods can be used to identify the direction of the incoming voice.

The display control unit 26 performs control to display the virtual object superimposed on the actual scenery within the user's field of view that is transmitted through the display unit 12 . This virtual object includes three-dimensional illustrations, figures, characters, icons, etc., and indicates the direction of arrival of the sound, the volume, type, etc. of the sound. The display control unit 26 changes the type of sound indicated by the virtual object according to the user's hearing level and displays it on the display unit 12. When the hearing level is high (the degree of hearing loss is low and it is possible to hear relatively well), the display control unit 26 may, for example, change the color of the virtual body displayed depending on the person speaking or the pitch of the voice, or change the color depending on the volume of the voice. Change the size of the virtual body or change the shape of the virtual body depending on the tone of voice (emotion). On the other hand, if the hearing level is low (the degree of hearing loss is high and it is difficult to hear), the display control unit 26 controls the size of the virtual object to be displayed depending on the type of sound (for example, a car, a bicycle, an alarm, etc.). Change the shape, etc. Furthermore, the display control unit 26 displays the virtual object including the direction from which the sound came. For example, the display control unit 26 may cause the virtual object to appear on the display unit 12 from the direction in which the sound arrives, or may divide the area around the user into 12 equal parts with the front of the field of view in the 12 o'clock direction, and The direction of arrival may be displayed as "○ o'clock direction".

The communication control unit 27 performs various communications with a server device (not shown) via the communication unit 13. For example, a server device may be configured to include at least some of the functions of the voice content recognition unit 23, emotion analysis unit 24, and voice direction identification unit 25, and various information executed by the server device may be acquired.

The hearing level determining unit 28 determines the hearing level (also referred to as hearing loss level) of the user who uses the sound generation notification device 10. In general, the hearing level differs depending on the user, with ``mild hearing loss'' making it difficult to hear small voices, ``moderate hearing loss'' making it difficult to hear normal conversations, ``severe hearing loss'' making it difficult to hear normal conversations, and ``severe hearing loss'' making it difficult to hear people talking close to their ears. In some cases, it has been categorized as ``hearing loss.'' In this embodiment, the sound generation notification device 10 changes the type of sound indicated by the virtual object displayed on the display unit 12, among sounds generated in the surroundings, according to the user's hearing level. In other words, for users with "severe hearing loss" and "severe hearing loss" who are unable to hear normal conversations, when a highly dangerous sound or a sound that calls for attention occurs, a virtual object indicating the occurrence of this type of sound is displayed. Display. Furthermore, for users with "mild hearing loss" and "moderate hearing loss" who have difficulty hearing small voices, a virtual object indicating the occurrence of voices (sounds) during conversation is displayed.

The hearing level determination unit 28 performs a hearing determination, for example, when the sound generation notification device 10 is used for the first time. For example, the hearing level determination unit 28 is communicatively connected to the television used by the user via the communication unit 13 to obtain the user's set volume value (dB). Then, the hearing level determination unit 28 determines the hearing (hearing loss) level based on how much higher the user's set volume value is than the general TV volume value (dB) specified by the TV manufacturer or the like. Further, the hearing level determination unit 28 may determine the user's hearing level using other methods described below.

A: The TV stores in advance the TV volume value (dB) data of the general viewer who is watching the TV on a server device on the network. For example, the hearing level determination unit 28 is communicatively connected to the television used by the user via the communication unit 13 to obtain the user's set volume value (dB). Then, the hearing level determining unit 28 may determine the hearing level based on how much the user's set volume value differs from the average volume value of the general audience on the server device.

B: The sound generation notification device 10 to which headphones or earphones are attached may be equipped with an audiometer function, and the user may perform a hearing test upon first use and determine the hearing level based on the results. Furthermore, instead of the audiometer function, the sound generation notification device 10 may be equipped with a bone conduction receiver function to determine the hearing level through a bone conduction test.

C: When setting up for the first time, the sound notification device 10 attached to headphones or earphones will emit a setting guidance voice, and the user can set the sound notification device 10 using voice recognition by answering “yes” or “no”. be able to do so. In this case, the hearing level may be determined based on the volume value of the setting guidance voice. Note that a user with severe hearing loss can set the setting by displaying the setting guide in text on the display unit 12 of the sound generation notification device 10 after the maximum sound level of the setting guide voice has finished.

D: The sound generation notification device 10 may be equipped with an auditory brainstem response test (ABR) function using brain waves, and the hearing level may be determined while the user is sleeping or at rest. Furthermore, the user himself or herself may set the hearing level.

The storage unit 29 stores various control data and control programs for the sound generation notification device 10. In this embodiment, the storage unit 29 stores control programs for respectively executing the three-dimensional figure projection mode and the danger/warning mode. The three-dimensional figure projection mode mainly supports difficulties in sound source localization, and is, for example, a control program for users with "mild hearing loss" and "moderate hearing loss." Further, the danger/warning notification mode is for notifying dangers and cautions occurring in the surroundings, and is a control program for users with "severe hearing loss" and "severe hearing loss", for example. The storage unit 29 also stores the user's hearing level based on the hearing ability determination.

Next, the operation of the sound generation notification device 10 will be explained. The sound generation notification device 10 automatically executes the three-dimensional figure projection mode or the danger/warning mode depending on the determined hearing level of the user. FIG. 2 is a flowchart showing the operation procedure of the sound generation notification device in the three-dimensional figure projection mode. FIG. 3 is a diagram showing an example of a virtual object displayed in a group meeting in which the three-dimensional figure projection mode is executed. In the example of FIG. 3, it is assumed that the user is participating in a group meeting (conference) with five members P1 to P5.

First, the sound generation notification device 10 acquires ambient sound collected by the sound collection unit 11 (step S1). More specifically, the audio acquisition unit 21 acquires multiple channels of audio signals output by the sound collection unit 11. This acquired acoustic signal is output to the sound source separation section 22.

Next, the sound generation notification device 10 uses the sound source separation unit 22 to separate the plurality of acoustic signals included in the plurality of channels into acoustic signals for each sound source (step S2). The sound source separation unit 22 separates, for example, a plurality of acoustic signals of a plurality of people having a conversation at the same time into acoustic signals of each person. Further, the sound source separation unit 22 can separate the sound signals of each sound source (such as the sound of a car running or a bicycle bell) from a plurality of sound signals in a city area where cars and bicycles come and go, for example.

Next, the sound generation notification device 10 uses the speech content recognition unit 23 to perform speech recognition processing on each sound signal whose sound source has been separated (step S3). More specifically, the speech content recognition unit 23 converts, for example, an acoustic signal of a speech uttered by a person into text data. The audio content recognition unit 23 also recognizes, for example, sounds other than those uttered by a person (for example, the sound of a car running, a warning sound, a bicycle bell or brake sound, various warning sounds, the sound of rain, the sound of wind, the sound of thunder, etc.). ) can be recognized by determining the type of voice.

Next, the sound generation notification device 10 determines whether the volume of the acquired sound is greater than a predetermined threshold (step S4). The sound generation notification device 10, for example, acquires the voice uttered by any of the five members P1 to P5, and determines whether the volume of this voice is greater than a predetermined threshold. In this determination, if the volume of the audio is not greater than the predetermined threshold (step S4; No), the process returns to step S1, and if the volume of the audio is greater than the predetermined threshold (step S4; Yes), The process moves to step S5.

Next, the sound generation notification device 10 uses the emotion analysis unit 24 to analyze the magnitude, emotion, pitch, etc. of the voice (step S5). More specifically, the emotion analysis unit 24 determines whether the speaker's emotion is positive (calm, happy, etc.) or negative (angry, sad, etc.) based on the volume and tone of the voice in the acoustic signal of the uttered voice. fear, etc.).

Next, the sound generation notification device 10 uses the audio direction identification unit 25 to identify the direction of arrival of the audio (step S6). More specifically, the audio direction identification unit 25 calculates the difference in arrival time of each of the collected audio signals of the plurality of channels, and identifies the direction of arrival of the collected audio based on this time difference.

Next, the sound generation notification device 10 causes the display control unit 26 to generate a virtual object based on the content of the uttered voice, emotion, etc. (step S7), and displays this virtual object in a superimposed manner on the display unit 12. (Step S8). As shown in Figure 3, in a group meeting, for example, in a scene where member P2 is speaking normally and member P4 is speaking happily, member P2's emotions are calm and member P4's emotions are happy. It is analyzed as Therefore, the display control unit 26 displays the virtual object 100 representing peace over the head of the member P2, and displays the virtual object 101 representing happiness over the head of the member P4. At this time, the display control unit 26 further displays information 110 and 111 indicating the direction of arrival of the audio near the virtual objects 100 and 101, respectively. In the example of FIG. 3, information 110 indicating "11 o'clock direction" is displayed near the virtual object 100 of member P2, and information 111 indicating "1 o'clock direction" is displayed near the virtual object 101 of member P4. . Note that the information indicating the direction of arrival of the audio and the virtual object may not be displayed separately, but the information 110 indicating the direction of arrival of the audio may be included in the virtual object 100 and displayed. Similarly, information 111 indicating the direction of arrival of audio may be included in the virtual object 101 and displayed.

With this configuration, by relying on the displayed virtual bodies 100 and 101, it is possible to recognize who is speaking and where, and thus it becomes possible to localize the sound source. Furthermore, since the virtual bodies 100 and 101 also display the emotions of the speaker, it becomes possible to intuitively understand what kind of person is speaking and the person's characteristics, personality, emotions, etc. Furthermore, even if someone suddenly speaks to you from a direction out of your field of vision, you can intuitively understand the characteristics, personality, and emotions of the person who is speaking to you, so you can instantly sense whether or not there is danger, for example.

The sound generation notification device 10 repeatedly executes the processing from steps S1 to S8, and ends the processing when the user operates a stop switch, for example.

Next, the operation of the sound generation notification device 10 in the danger/warning notification mode will be explained. FIG. 4 is a flowchart showing the operation procedure of the sound generation notification device 10 in the danger/warning notification mode. FIG. 5 is a diagram showing an example of a virtual object displayed in a city area where the danger/warning notification mode is executed. In the example of FIG. 5, it is assumed that the user is moving around the city by walking. In addition, in the flowchart of FIG. 4, steps that overlap with those of the flowchart of FIG. 2 are given the same reference numerals, and the description thereof will be omitted.

When the sound generation notification device 10 performs voice recognition processing on each sound signal whose sound source has been separated by the voice content recognition unit 23 (step S3), the sound generation notification device 10 determines whether the recognized voice is dangerous or not. It is determined whether the sound is high pitched or calls for attention (step S4a). Here, the highly dangerous sounds and the sounds that call for attention include, for example, car horn sounds, bicycle bell and brake sounds, various alarm sounds, human screams, heavy rain, strong winds, thunder, and other sounds. If the recognized voice is not a voice that is highly dangerous or calls for attention (step S4a; No), the sound generation notification device 10 returns the process to step S1, and the recognized voice is not a voice that is dangerous or calls for attention. If the sound is highly sensitive or calls for attention (step S4a; Yes), the process moves to step S5a.

Next, the sound generation notification device 10 analyzes the type of sound using the sound content recognition unit 23 (step S5a). More specifically, the sound generation notification device 10 analyzes the type of the recognized voice (car horn sound, bicycle bell and brake sound, various alarm sounds, human screams, rain sound, wind sound, thunder sound). and separate them by type.

Next, the sound generation notification device 10 uses the display control unit 26 to generate a virtual object based on the type of sound (step S7), and displays this virtual object in a superimposed manner on the display unit 12 (step S8). As shown in Figure 5, while walking in a city area, for example, if a bicycle approaches from between houses in front that are difficult to see directly and rings a bell, the sound will be of the type alarm. , the display control unit 26 displays a virtual object 102 indicating a bicycle on the road superimposed on the display unit 12, and displays information 112 indicating the arrival direction of the voice "2 o'clock direction" beside the virtual object 102. . As a result, the user can understand that a bicycle is approaching from the 2 o'clock direction, that is, from the right front, and can prevent contact with the bicycle by yielding to the bicycle. Furthermore, not only will it be possible to identify the type of sound source and localize the sound source, but it will also be possible to immediately detect danger even when no sound can be heard. In the example of FIG. 5, the virtual object 102 representing a bicycle is displayed, but for example, in a scene where a nearby person is shouting words such as "danger" or "run away", voice-recognized text data is displayed. It is preferable to display "danger" and "run away" as virtual entities on the display unit 12 together with information on the direction of arrival. Note that the operation of the danger/warning mode can be used not only by people with severe to severe hearing loss, but also by people with normal hearing when listening to music with earphones, for example.

The sound generation notification device 10 repeatedly executes the processing from steps S1 to S8, and ends the processing when the user operates a stop switch, for example.

As described above, the sound generation notification device 10 according to the present embodiment includes the audio acquisition unit 21 that acquires collected audio, the audio direction identification unit 25 that identifies the direction of arrival of the audio, and the audio content that recognizes the content of the audio. A recognition unit 23, a display unit 12 that displays an image, and a display control unit 26 that causes the display unit 12 to display images of the virtual objects 100 to 102 generated based on the arrival direction and content of the audio. Since the user can recognize the direction and content of the sound through the virtual bodies 100 to 102, the user can easily localize the sound source and appropriately notify the occurrence of the sound.

Furthermore, the sound generation notification device 10 according to the present embodiment includes an emotion analysis unit 24 that analyzes the speaker's emotion contained in the voice, and a display control unit 26 that can detect the arrival direction of the voice, the content of the voice, and the loudness. In order to display on the display unit 12 images of the virtual bodies 100 to 101 that are generated based on the person's character and emotion, it is possible to intuitively understand, for example, what kind of person is speaking and the person's characteristics, personality, and emotion. Be able to understand.

In addition, the sound generation notification device 10 according to the present embodiment includes a hearing level determining unit 28 that determines the hearing level of the user, and the display control unit 26 controls the display unit 12 to display the information displayed on the display unit 12 according to the determined hearing level. In order to change the type of sound indicated by the virtual objects 100 to 102, for example, when a highly dangerous sound or a sound that calls for attention is generated for a user with "severe hearing loss" or "severe hearing loss", Also, a virtual object corresponding to this audio type is displayed. This allows the user to not only identify the type of sound source and localize the sound source, but also to immediately sense danger even when no sound is heard.

In addition, in the sound generation notification device 10 according to the present embodiment, the display unit 12 is a glass monitor that is placed in front of the person's eyes and displays an image that overlaps the transmitted field of view. You can easily notify the user that a sound has occurred in the surrounding area.

Each component of the illustrated sound generation notification device 10 is functionally conceptual, and does not necessarily have to be physically configured as illustrated. In other words, the specific form of each device is not limited to what is shown in the diagram, and all or part of it may be functionally or physically distributed or integrated into arbitrary units depending on the processing load and usage status of each device. It's okay.

The configuration of the notification control device 20 of the sound generation notification device 10 is realized by, for example, a program loaded into a memory as software. The above embodiments have been described as functional blocks realized by cooperation of these hardware or software. That is, these functional blocks can be implemented in various forms using only hardware, only software, or a combination thereof.

This disclosure includes matters that contribute to the realization of "health and well-being for all" in the Sustainable Development Goals (SDGs) and contribute to value creation through healthcare products and services.

10 Sound generation notification device 11 Sound collection unit 12 Display unit 20 Notification control device 21 Audio acquisition unit 22 Sound source separation unit 23 Audio content recognition unit 24 Emotion analysis unit 25 Audio direction identification unit 26 Display control unit 28 Hearing level determination unit 100, 101 , 102 Virtual object 110, 111, 112 Information indicating the direction of arrival of audio

Claims (5)

an audio acquisition unit that acquires the collected audio;
a voice direction identification unit that identifies the direction of arrival of the voice;
a voice content recognition unit that recognizes the content of the voice;
a display section that displays an image;
A sound generation notification device comprising: a display control unit that causes the display unit to display an image of a virtual object generated based on the arrival direction and content of the sound. comprising an emotion analysis unit that analyzes the speaker's emotion contained in the voice,
The sound generation notification according to claim 1, wherein the display control unit causes the display unit to display an image of the virtual object generated based on the arrival direction of the sound and the content, size, and emotion of the sound. Device. comprising a hearing level determination unit that determines the user's hearing level;
The sound generation notification device according to claim 1 or 2, wherein the display control unit changes the type of the sound indicated by the virtual object displayed on the display unit, depending on the determined hearing level. The sound generation notification device according to claim 1 or 2, wherein the display unit is a glass monitor that is placed in front of the person's eyes and displays an image that overlaps the transmitted field of view. an audio acquisition step of acquiring the collected audio;
a voice direction identification step of identifying the direction of arrival of the voice;
a voice content analysis step of analyzing the content of the voice;
A sound generation notification method comprising: a display control step of displaying on a display unit an image of a virtual object generated based on the arrival direction and content of the sound.

Images