RU2345422C2 - Imaging method and device intended for speech sound propagation detection - Google Patents

Abstract

FIELD: physics, acoustics.

SUBSTANCE: invention refers to speech sound propagation detectors and deaf-mute spatial orientation facilities. Sound propagation is displayed relative to direction of listener's direct sight on colour displays by means of vertical indicating strips of width varying depending on sound propagation change. Indicating strips travel across displays field relative to neutral position depending on sound propagation from above or from below with respect to patient's head. Colour of vertical indicating strips varies depending on speech distribution propagation change in two potential opposite directions: in front or behind. The device contains eyeglass frame with left and right eyepieces, two ear arms with mounted microphones, left and right speech perception and imaging channels, speech sound identification unit, left and right colour liquid crystal displays. The central microphone is mounted by means of ball-and-socket joint on eyeglass frame bridge.

EFFECT: development of speech sound propagation detector and deaf-mute spatial orientation facility.

2 cl, 3 dwg

Description

The invention relates to technical means of speech recognition, namely recognition of the direction of speech sound propagation, and is intended to ensure the life of deaf and hard of hearing people. Speech recognition is provided by forming colorographic images, which are a function of speech sounds. The invention can be used as a technical tool for the training of deaf-mute people in the development of colloquial speech.

In a previously filed application No. 2004139176/12 (042581), the authors set forth a method and design of a device for recognizing the meaning of speech by deaf and dumb people. The effectiveness of the use of this device will largely depend on how accurately a person will recognize with the help of this device, if there are several interlocutors, which of them is currently talking. This can be done only on condition that this device technically implements a reliable method of visual recognition of the direction of speech sound propagation. Recognition of the direction of sound propagation is very important for a deaf-mute person at the time of the sound of an alarm: the signal of an approaching car or train, the sound of a flying airplane, the signal of a cargo crane at a construction site and so on. The direction of sound propagation relative to the human head can be any, and a deaf-mute person must reliably visually recognize it.

In a previously filed application No. 2004139176/12 (042581), the authors set forth a method for recognizing sound in two directions relative to the listener's head (left and right) through the use of left and right image forming channels for speech recognition. However, this is not enough to recognize the direction of sound propagation from the front, back, top and bottom. To determine all possible directions of sound propagation, it is necessary to have at least three microphones spaced in space at a finite distance between each other.

The present invention relates to technical means for recognizing the direction of sound propagation by visualizing the colorographic symbols of the direction of sound propagation on a color screen almost synchronously with the development of speech.

Closest to the proposed invention is a method of forming an image on the screen depending on the direction of propagation of speech sound, described in [1] (US patent 6629076 B1, published September 30, 2003). In the description of this device in Fig. 5 it is shown that the direction of sound propagation is indicated on the display in the form of arrows 64. Arrows 64 allow you to indicate the direction of propagation of sound waves lying in only one plane - the plane of the location of the vectors of maximum sensitivity of the petal diagrams of sound Creation of all microphones 12 located on the frame and the temples of the glasses. From the description of the invention and the location of the microphones 12 shown in Fig. 1, it follows that with such an arrangement of the microphones 12 it is impossible to identify the direction of the sounds coming from above and below with respect to the human head since the claimed device does not have microphones whose petal diagram of sound sensitivity would be directed upward in relation to a person’s head. This is a significant drawback of the claimed device. In addition, the proposed method of visualization using arrows 64 (Fig. 5) does not allow visualization of all spatial directions of sound propagation, since arrows 64 are always located in the same plane and display the direction of the sound propagation vector in only one plane of the X and Y coordinates, which is insufficient for displaying the spatial direction of the sound propagation vector, for which three coordinates are necessary: X, Y, Z. Differences in the proposed method for recognizing the sound propagation direction speech by the known method described in [1]:

1. The proposed method allows to determine and visually recognize the direction of propagation of speech sounds along each of the three spatial coordinates X, Y, Z.

2. Recognition of the propagation direction of speech sounds is carried out by forming a color vertical indicator strip on the color screen that changes its width, position and color depending on the change in the propagation direction of speech sounds along each of the three spatial coordinates X, Y, Z.

3. A central microphone mounted on a ball joint on the nose bridge is introduced into the design of the device, which allows implementing this method of recognizing the direction of propagation of speech sounds, so that this microphone and, therefore, its lobe diagram of sound sensitivity together with the microphone can be manually moved to any direction within a given spatial solid angle, including an upward direction with respect to the human head.

The basis of the device for recognizing the direction of propagation of speech sounds is a device for recognizing the meaning of speech, proposed and described by the authors of this application in the previously filed by the same authors application No. 2004139176/12 (042581).

The technical result of the invention is to increase the information content of sound information perceived by a deaf and hard of hearing person.

The specified technical result is achieved by the fact that in the known method of image formation for recognizing the direction of propagation of speech sound, including converting sound through microphones into sound signals, automatically adjusting the level of sound signals, identifying sound signals, displaying sound signals on the left and right color liquid crystal displays, according to the invention, the direction of sound propagation is displayed relative to the direction of direct view of the listener on the left color by means of the left vertical indicator strip located along the left vertical side of the left display and on the right display by the right vertical indicator strip located along the right vertical side of the right display, the width of these vertical indicator stripes changing opposite to each other depending on the change in the direction of sound propagation: right or left, in addition, the vertical indicator left and right stripes move along the display field relative to the medium of the position indicated on the screens by vertical fixed marks, up or down depending on whether the sound of speech comes from above or below in relation to the person’s head, the color of the vertical indicator bars changes depending on the change in the direction of speech propagation in two possible opposite directions: front or back.

The specified technical result is also achieved by the fact that the known image-forming device for recognizing the direction of the speech sound propagation, containing a spectacle frame with left and right eyepieces, two earhooks with microphones installed on each of them, left and right channels of speech perception and image formation, identification unit speech sounds, the left and right color liquid crystal displays according to the invention have a central microphone mounted by means of a ball joint on the nose bridge you points.

Consider an image forming method for recognizing the direction of propagation of speech sounds. Figure 1 on the field of the right display 1 presents an image of a group of color symbols 2, the essence of which is disclosed in the application No. 2004139176/12 (042581). On the field of the right display 1 is also shown a moving vertical indicator strip 3 located along the right vertical side of the display field, and a fixed pointed mark 4, which serves to indicate the middle position of the strip 3. On the field of the strip 3, strictly moving in the vertical direction is the moving mark 5, which moves up and down together with the vertical indicator strip 3. By the ratio of the relative positions of the movable 5 and fixed 4 marks, the person determines where the vertical indicator strip 3 has moved - up and and down. On the left display, located on the left eyepiece of the glasses, a colored vertical indicator strip and a pointed mark are located along the left side of the left display field.

In life practice, a person has to repeatedly orient his hearing aid (a pair of ears) in the direction of his maximum sensitivity to a sound source in order to most accurately identify the nature of the disturbing sound or to accurately identify the meaning of speech. For these purposes, a universal bodily organ is attached to a person - the neck of a person. The neck allows you to mobilely orient the head of a person and, therefore, the organs of hearing and vision in the right direction. She plays the role of a kind of hinge. The apple of the human eye also plays the role of a ball joint.

In technical devices, ball joints are often used where it is necessary to orient a stream of water, air or a stream of light in the desired direction. Ball joints are used in the design of floodlights, antennas, air fans, studio manipulators with microphones mounted on them. Correspondents of radio and television always orient their microphones in the direction of maximum sound power of the voice of the talking person. It follows that it is very advisable to use microphones using hinges.

The technical device by means of which a colorographic image of speech sounds and colored vertical indicator strips is formed is implemented using specialized glasses presented in figure 2.

This device has a frame 1, two eyepieces 2 and 3, two color liquid crystal displays 4 and 5, located in the upper zones of the eyepieces of glasses; two corrective lenses 6 and 7, superimposed on the plane of two color liquid crystal displays from the side of the observer's eyes; two earhooks 8 and 9, two side microphones 10 and 11, the central microphone 12, and the axis of the maximum sound sensitivity of the microphone 12 can move within the solid angle 14 by installing it on a ball joint 13; a unit for identifying speech sounds (BIZ) 15 with a two-channel transceiver radio device.

For stereophonic perception of sound, nature is given two hearing aids (ears), spaced in space at a distance approximately equal to the diameter of the head. When sound propagates, a sound wave, sequentially in time, first reaches the hearing aid of one ear, and then with delay (with a phase shift) and reduced sound pressure reaches the hearing aid of the other ear. According to the magnitude of the delay in the propagation of the sound wave and how the sound power in the other ear decreases, the corresponding structures of the human brain fix the direction of the propagation of speech sound in relation to the human head, thereby realizing a stereo effect.

In the same way, in the technical device proposed by the authors, for the realization of the stereo effect, it is necessary to have several microphones simultaneously, allowing to perceive sound along the three spatial coordinates X, Y, Z at the same time. From a technical point of view, the implementation of the stereo effect and, therefore, the determination of the direction of sound propagation in space is possible with three microphones 10, 11, 12 spaced in space, as shown in FIG. 2. The petal diagram of the sound sensitivity of the left microphone 10 is directed to the left with respect to the human head. For the right microphone 11, it is directed to the right. For the central microphone 12, the axis of maximum sound sensitivity can move within the solid angle 14 by installing it on the ball joint 13, and it can also be directed up and forth with respect to the human head due to the joint 13.

At the same time, microphones 10, 11, 12 will perceive sound waves coming in the front and back directions with respect to the human head, but microphones 10, 11 will do this with less sensitivity, which does not interfere with identifying these directions.

The central microphone 12 provides the perception of speech sounds coming from above and from the front (when turning it in the “forward” direction), with greater sensitivity and lower with less sensitivity with respect to a pair of microphones 10, 11. This fact allows us to determine the direction of speech sound propagation as above , and from below in relation to the human head. Since this microphone 12 is mounted on the nose bridge of the spectacle frame and is shifted forward by a certain distance with respect to the side microphones 10 and 11, this makes it possible to determine the direction of sound coming from the front or rear using it and the side microphones 10, 11 relation to the human head.

The mounting of the central microphone 12 on the ball joint 13 allows the deaf-mute person to manually manually change the orientation of the microphone 12 and thereby choose the optimal mode of perception of speech sound. This is very important to do in the presence of noise that violates the clear perception of speech sounds.

The speech sound identification unit 15 is common to the left and right channels. Through this unit, a spectral analysis of speech sound signals is carried out, speech sounds are identified, for example, by the method described in [3] - [9], and the direction of speech sound propagation is determined. The exchange of information between block 13 and electronic devices located in the earhooks 8 and 9 is carried out over the air.

The technical device for image formation for recognizing the direction of propagation of the sound of speech, presented in figure 2, has three independent channels for speech perception and image formation: the left channel of the image formation (LKFI), the right channel of the image formation (PKFI), the central channel (CC), serving to determine the direction of speech sound propagation with maximum sensitivity within the solid angle 14 and with reduced sensitivity in all directions outside the given angle. The left channel (LKFI) includes a left microphone 11, an electronic control device located in the left earhook 8, the left color liquid crystal display 4 and the left correction lens 6.

The right channel for imaging (PCFI) of speech sounds includes, respectively, a right microphone 11, an electronic control device located in the right earhook 9, a right color liquid crystal display 5 and a right correction lens 7. The central channel (CC) includes a central microphone 12 and an electronic control device located in the right earhook 9.

Three independent channels (LKPF, PKFI, CC) for speech perception and image formation also allow you to realize a stereo effect and thereby increase the orientation of a deaf-mute person in space by observing on two displays two moving vertical indicator bars indicating the direction of sound propagation relative to the head of a deaf-mute person .

The movement of the vertical indicator strip 3 (see Figure 1) up from the middle position, marked 4, along the ordinate axis means that the sound of speech comes from above relative to the person’s head. Moving the vertical indicator strips 3 (see Figure 1) down from the middle position, marked 4, along the ordinate axis means that the sound of speech comes from below relative to the person’s head. The distance between the fixed mark 4 and the moving mark 5 is proportional to the strength of the sound coming from above or below in relation to the person’s head. Increasing the width of the right indicator strip on the right display and simultaneously reducing the width of the left indicator strip on the left display means that the sound of speech comes to the right of the person. Increasing the width of the left vertical indicator strip on the left display and simultaneously reducing the width of the right vertical indicator strip means that the sound of speech comes to the left of the person. If the sound comes in front of the person’s head, then both (left and right) indicator strips take one color, adopted only for the “front” direction. If the sound comes from behind the head of the listener, then both (left and right) indicator strips change color to another, adopted only for the direction of "back". A method and apparatus for image formation for recognizing the direction of propagation of a sound signal, including two independent image forming channels and a central channel, are implemented using the block diagram shown in FIG. 3.

An imaging device for recognizing the direction of propagation of a sound signal includes a left channel for image formation (LKFI) 1, all of whose elements are located in the left earhook of the observer's glasses, a right channel for image formation (PCFI) 2, a central channel (CC) 28, all of which are located in right earhook glasses, and sound identification block (BIZ) 3, which is located in the breast pocket of a person’s shirt. Figure 3 LCFI, PKFI, CC and BIZ circled by dashed lines. A description of the operation of the left (LCFI) 1 and right (PKFI) 2 channels is discussed in detail in the application No. 2004139176/12 (042581), filed by the authors earlier. It should be noted once again that according to the set of signals supplied to the inputs 1 and 2 of the controller 15, the direction of sound propagation relative to the head of a deaf-mute person is determined by the program embedded in the controller 15: from the right or from the left. This direction, as indicated earlier, is displayed on the displays in the form of colored vertical indicator strips 3 (see Figure 1), changing their width.

The mutually opposite change in the width of the vertical indicator stripes on the left and right displays is precisely the way to visualize the stereo effect when perceiving the sound of speech.

The central channel (CC) 28 includes: a central microphone (M3) 29, an amplifier (U3) 30, a bandpass filter (PF3) 31, a link for automatically adjusting the sound signal level (AGC3) 32, an analog-to-digital converter (ADC3) 33, a delay link by time (ЗЗ 3) 34. All radioelements (CC) 28 are located in the right earhook of points.

In accordance with the structural diagram shown in FIG. 3, for the central channel (CC) 28, the microphone output (M3) 29 is connected to the input 1 of the amplifier (U3) 30, the output of the amplifier 30 is connected to the input of the bandpass filter (PF3) 31 and with the input (ADC3) 33. The output of the band-pass filter 31 is connected to the input of the link (AGC3) 32. The output of the link (AGC3) 32 is connected to the input 2 of amplifier (U3) 30. The output (ADC3) 33 is connected to the input of the delay link 34, the output of the delay link 34 connected simultaneously with two inputs: input 3 of the controller (КР2) 25 and input 3 of the controller (КР1) 10. Central channel (CC) 28 operation em as follows. The sound of speech is perceived by the microphone 29, is converted into a speech sound signal, which is fed to the input 1 of the normalizing amplifier 30. The amplifier 30 is used to control the value of the sound signal level at the output of this amplifier 30 with a possible significant change in sound power. For this, a special link for automatic adjustment of signal level 32 is used. By means of a band-pass filter 31, the signal of the most common and stable formants is extracted from the sound signal, and through this signal the link (AGC3) 32 is controlled. The output signal of link 32 automatically changes the coefficient at input 2 of link 30 the gain of the amplifier (U3) 30, thereby automatically being kept, normalized within the range of technically justified, the signal level at the output of amplifier 30. From the output of amplifier 30, the normalized signal stays at the input (ADC3) 33, through which the digitization of the input analog signal. The output digital signal of speech sound from the output (ADC3) 33 is fed to the input of the time delay link 34. From the output of the link 34, the output signal is sent simultaneously to the inputs of two links: input 3 of link 10 and input 3 of link 25. With the appearance of an audio signal at the output of the central microphone (M3) 29, the values of the digital codes at the inputs of 3 links 10, 25 change and, therefore, are shifted up or down relative to the middle position marked 4, the colored vertical indicator bars on the screens of the left and right displays, signaling that the sound comes from above or below. Link 34 performs the same functions as links 9 and 24. The sound identification unit 3 has two independent branches of processing sound signals, moreover, the controller 15 provides the operation of the LCFI, and the controller 16 provides the work of the PCFI. Transmitters 13 and 17, as well as receivers 14 and 18 are two-channel, with frequency or time separation in the operation of two channels.

List of drawings

Figure 1 - Colorographic image of sound signals on a color screen, for example, the word "PRO-SHU"

Figure 2 - An image forming apparatus for recognizing the meaning of speech and the direction of propagation of speech sound.

Figure 3 - Block diagram of an image forming apparatus for recognizing the meaning of speech and the direction of propagation of speech sound.

Information confirming the possibility of carrying out the invention.

The present invention can be implemented using a device whose design is shown in Figure 2 and a structural diagram of which is shown in Figure 3. Elements of the device can be made using analog and digital microcircuits, small-sized studio radios and radio transmitters, digital controllers and color liquid crystal displays, which are widely used in digital cameras and cameras.

Literature

1. Patent US 6629076 B1, published September 30, 2003.

2. Application RU No. 2004139176/12 (042581), IPC G09B 21/00 (2006.01), G10L / 06 (2006.01).

3. Patent RU No. 2230375, IPC 7 G10L 15/00, 17/00, published in 2004, BI No. 16 (II part) p. 437.

4. Patent RU No. 2234746, IPC 7 G10L 19/02, published 2004.08.20.

5. RF patent No. 2047912, IPC G10L 7/06, published November 10, 1995.

6. RF patent №2161826, IPC G10L 17/00, published January 10, 2001.

7. US patent No. 6411930, IPC G10L 15/08, 06/25/2002.

8. US patent No. 5995927, IPC G10L 9/00, 11/30/199.

9. US patent No. 6389392, IPC G10L 17/00, 05/14/2002.

Claims (2)

1. An image forming method for recognizing a sound propagation direction of a speech, including converting sound through microphones into sound signals, automatically adjusting the sound signal level, identifying sound signals, displaying sound signals on the left and right color liquid crystal displays, wherein the sound propagating direction is displayed relative to the direct gaze of the listener on the left color display by means of the left vertical indicator strip, lying along the left vertical side of the left display and on the right display by means of a right vertical indicator strip located along the right vertical side of the right display, and the width of these vertical indicator stripes changes opposite to each other depending on the change in the direction of sound propagation: right or left, while vertical indicator left and right strips move along the field of displays relative to the middle position, marked on the screens as vertical motionless marks, up or down, depending on whether the sound of speech comes from above or below in relation to the person’s head, the color of the vertical indicator stripes changes depending on the change in the direction of speech sound propagation in two possible opposite directions: front or back. 2. An imaging device for recognizing the direction of propagation of speech sound, containing a spectacle frame with left and right eyepieces, two earhooks with microphones installed on each of them, left and right channels of speech perception and image formation, speech sound identification block, left and right color liquid crystal displays, characterized in that it has a central microphone mounted by means of a ball joint on the nose bridge of the spectacle frame.

Images