JP7000547B1 - Programs, methods, information processing equipment, systems - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the convenience of voice recognition processing. SOLUTION: The program is to be executed by a computer including a processor and a memory, and the program is at least one from a step of acquiring a sound collected by a sound collector and the acquired sound. A step of extracting one sound, a step of converting the extracted sound into text information, a step of estimating the role of the speaker of the extracted sound based on the text information, and a step of converting the converted text information. A program that executes a step that presents the role to the user in an identifiable manner [selection diagram] Fig. 4

Description

The present disclosure relates to programs, methods, information processing devices and systems.

A technique for distinguishing speakers by the characteristics of voice waveforms is known. For example, Patent Document 1 describes that a user is authenticated by using biometric information including voice information.

Japanese Patent Application Laid-Open No. 2015-061086

However, in the conventional system, the speaker cannot be identified unless the voice information is registered in advance. Therefore, even if the text information is generated by the voice recognition process, if the voice information is not registered in advance, the speaker cannot be identified, and the convenience of the voice recognition process may be impaired.

An object of the present disclosure is to improve the convenience of speech recognition processing.

According to one embodiment, a program for causing a computer including a processor and a memory to execute the program, from the step of acquiring the sound collected by the sound collector to the processor and the acquired sound. A step of extracting at least one sound, a step of converting the extracted sound into text information, a step of estimating the role of the speaker of the extracted sound based on the text information, and the converted text information. Is provided with a step of presenting the role to the user in an identifiable manner, and a program to execute.

According to the present disclosure, the convenience of voice recognition processing can be improved.

It is a figure which shows the whole structure of the system 1. It is a figure which shows the functional configuration of a server 20. It is a figure which shows the data structure of the text information database 2021 and the voice information database 2022 stored in a server 20. It is a figure which shows the outline of the apparatus which comprises the system 1. 6 is a flowchart showing a series of processes when the server 20 generates text data based on sound data. It is a figure which shows the display example of the text data generated based on the conversation between a surgeon and an assistant. It is a figure which shows the display example of the text data generated based on the conversation between a speaker and a viewer. It is a figure which shows the display example of the text data generated based on the conversation between a manager and a worker. It is a figure which shows the whole structure of the system 1A in the 2nd Embodiment. It is a figure which shows the functional configuration of the server 20A in the 2nd Embodiment. It is a figure which shows the data structure of the image information database 2023 which the server 20A stores in the 2nd Embodiment. It is a figure which shows the outline of the apparatus which comprises the system 1A in the 2nd Embodiment. 6 is a flowchart showing a series of processes when the control unit 203A of the server 20A generates text data based on sound data and image data.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are designated by the same reference numerals. Their names and functions are the same. Therefore, the detailed description of them will not be repeated.

<First Embodiment>
<Overview>
In the following embodiment, the system 1 that estimates the role of the utterer and stores the utterance content of the utterer and the estimated role as text data will be described.

The system 1 collects ambient sound by a sound collecting device. The system 1 extracts at least one or more sounds from the sound data based on the collected sounds. The system 1 converts the utterance content of the extracted voice into text information. System 1 estimates the role of the speaker based on the text information. The system 1 stores text data in which the estimated role is added to the text information, and presents the text data in response to a request from the user.

The system 1 can be installed in, for example, a medical facility such as a hospital. Specifically, for example, a sound collecting device is installed in the operating room, and the system 1 converts conversations during surgery such as a surgeon and an assistant into text information and stores it together with a role estimated from the text information. Further, for example, a sound collecting device is installed in the hospital room to convert conversations between the attending physician and a nurse into text information and store it together with a role estimated from the text information. The surgeon and the attending physician are mainly examples of persons in charge of performing medical treatment, and assistants and nurses are examples of persons in charge of assisting. This makes it possible to store intraoperative and daily conversations as text data including text information and roles.

The system 1 can also be installed in a place such as a seminar or a press conference. Specifically, for example, a sound collecting device is installed in the venue, and the system 1 converts a question and answer session between the speaker and the viewer into text information and stores it together with a role estimated from the text information. This reduces the trouble of creating minutes. In addition, the content of the question can be easily reviewed. The speaker is an example of the main speaker, and the main speaker may be a person who controls the progress of the meeting, for example, a moderator.

Further, the system 1 may be installed at a work site or the like, for example. Specifically, for example, a sound collecting device is installed at the site, and the system 1 converts the contents of instructions from the manager to the worker, the contents of the report from the worker to the manager, etc. into text information, and texts. Memorize with the role estimated from the information. The administrator may be referred to as an instructor. In addition, the worker is an example of a managed person managed by the manager. This makes it possible to confirm whether or not there is an omission of instructions from the administrator to the worker when a trouble occurs.

<1 Configuration diagram of the entire system>
FIG. 1 is a diagram showing the overall configuration of the system 1.

As shown in FIG. 1, the system 1 includes a server 20, an edge server 30, and a sound collector 40. The server 20 and the edge server 30 communicate with each other via the network 80. The edge server 30 is connected to the sound collecting device 40. For example, the sound collector 40 is a transmission / reception device based on a communication standard used in a short-range communication system between information devices. Specifically, the sound collector 40 uses a 2.4 GHz band such as a Bluetooth (registered trademark) module to receive a beacon signal from another information device equipped with the Bluetooth (registered trademark) module. The edge server 30 acquires information transmitted from the sound collecting device 40 based on the beacon signal using the short-range communication. In this way, the sound collecting device 40 transmits the acquired voice information of the speaker to the edge server 30 by short-range communication without going through the network 80. The edge server 30 may be connected to the sound collecting device 40 via the network 80.

The server 20 manages information about sound. Information about sound includes, for example, sound data, text data generated based on voice extracted from sound, and the like. The server 20 shown in FIG. 1 has a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29.

The communication IF 22 is an interface for inputting / outputting signals because the server 20 communicates with an external device. The input / output IF 23 functions as an interface with an input device for receiving an input operation from the user and an interface with an output device for presenting information to the user. The memory 25 is for temporarily storing a program, data processed by the program, or the like, and is, for example, a volatile memory such as a DRAM (Dynamic Random Access Memory). The storage 26 is a storage device for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive). The processor 29 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

In the present embodiment, the case where the system 1 has the server 20 is described as an example, but the system 1 may be formed as an aggregate of a plurality of servers. The method of allocating the plurality of functions required to realize the system 1 according to the present embodiment to one or a plurality of hardware is in consideration of the processing capacity of each hardware and / or the specifications required for the system 1. Can be determined as appropriate.

The edge server 30 receives the signal transmitted from the sound collecting device 40, and transmits the received signal to the server 20. Further, the edge server 30 transmits the signal acquired from the server 20 to the sound collecting device 40. The signal acquired from the server 20 includes, for example, information for updating the settings of the sound collecting device 40. Although FIG. 1 shows an example in which the number of edge servers 30 is one, a plurality of edge servers may be accommodated in the system 1.

The sound collector 40 collects ambient sounds and converts them into, for example, digital sound data. The sound collecting device 40 transmits a sound signal based on the sound data to the edge server 30. The sound collecting device 40 is realized by, for example, a microphone. The microphone is, for example, a directional microphone or an omnidirectional microphone. The directivity of the directional microphone may be unidirectional or bidirectional. The sound collecting device 40 is installed, for example, at a position where sound can be efficiently collected. Although FIG. 1 shows an example in which one sound collecting device 40 is used, a plurality of sound collecting devices 40 accommodated in the system 1 may be provided.

<1.1 Configuration of server 20>
FIG. 2 is a diagram showing a functional configuration of the server 20. As shown in FIG. 2, the server 20 functions as a communication unit 201, a storage unit 202, and a control unit 203.

The communication unit 201 performs a process for the server 20 to communicate with an external device.

The storage unit 202 stores data and programs used by the server 20. The storage unit 202 stores the text information database 2021 and the voice information database 2022 and the like.

The text information database 2021 stores text data generated based on the sound collected by the sound collector 40. Details will be described later.

The voice information database 2022 stores sound data based on the sound collected by the server 20 by the sound collecting device 40. Details will be described later.

The control unit 203 exerts a function shown as various modules by performing processing according to a program by the processor of the server 20.

The reception control module 2031 controls a process in which the server 20 receives a signal from an external device according to a communication protocol. For example, the reception control module 2031 controls the communication unit 201 and receives a sound signal transmitted from the sound collector 40 via the edge server 30.

The transmission control module 2032 controls a process in which the server 20 transmits a signal to an external device according to a communication protocol.

The acquisition module 2033 acquires sound data from the received sound signal. The acquisition module 2033 stores the acquired sound data in the voice information database 2022. The acquisition module 2033 stores the acquired sound data in the voice information database 2022, for example, when a predetermined requirement is satisfied. The predetermined requirements are, for example,:
-From the input of the recording start instruction to the input of the recording end instruction-Achievement of a preset time-Continuous generation of sound (for example, when sound is generated, recording is started and the sound is preset. Recording will stop if the period does not occur)

The voice analysis module 2034 analyzes the acquired sound data. The voice analysis module 2034 analyzes sound data, for example, when a predetermined requirement is satisfied. The predetermined requirements are, for example,:
-From the input of the recording (analysis) start instruction to the input of the recording (analysis) end instruction-Achievement of a preset time-Continuous generation of sound (for example, when sound is generated, analysis is started and analysis is started. Analysis will stop if no sound is generated for a preset period of time)

The voice analysis module 2034 extracts a predetermined voice from the acquired sound data. Specifically, the voice analysis module 2034 extracts a predetermined voice from the sound data based on, for example, any of the following information.
・ Characteristics of voice ・ Direction in which sound is collected ・ Timing when sound is collected ・ Sound collector that collects sound

More specifically, for example, the voice analysis module 2034 is a group consisting of voice features included in the sound data, for example, voice volume, pitch (frequency), voicedness, unvoicedness, phoneme type, formant, and the like. Analyze at least one selected from. The voice analysis module 2034 extracts voices estimated to have been generated by the same person based on the analysis results from the sound data.

Further, for example, when the sound collecting device 40 has directivity, the voice analysis module 2034 obtains the sound coming from the pointing direction from the sound data based on the directivity information of the sound collecting device 40. Extract.

Further, for example, when the timing of utterance is known in advance, the voice analysis module 2034 extracts the voice included in the sound data based on the time when the sound collecting device 40 collects the sound. For example, the voice analysis module 2034 refers to the schedule related to the utterance of a lecture or the like, and extracts the voice first uttered after the corresponding time is reached from the sound data.

Further, for example, when a plurality of sound collecting devices 40 are used, the voice analysis module 2034 extracts the sound collected by each sound collecting device 40 from the sound data.

Regarding the above extraction method, the voice analysis module 2034 may extract the voice of the speaker alone, or may extract the voice of the speaker by combining a plurality of methods.

Further, the voice analysis module 2034 converts the utterance content into text information by executing a voice recognition process on the extracted voice. As the speech recognition method, any existing method may be used. The converted text information is stored in the text information database 2021.

The estimation module 2035 estimates the role of the speaker based on the text information. For example, the estimation module 2035 estimates the role of the speaker by inputting text information into the trained model stored in the storage unit 202 of the server 20.

The trained model is generated, for example, by having a machine learning model perform machine learning according to a model learning program based on training data. In the present embodiment, the trained model is trained to output a role for a statement stored in, for example, the text information database 2021. At this time, for the learning data, for example, the character information about a predetermined remark is used as input data, and the role of the person who makes the remark is used as the correct answer output data. For example, the text information about the remark that leads the surgery is used as the input data, and the surgeon who is the role of the person who makes the remark that leads the surgery is used as the correct output data. In addition, the text information about the remarks that assist the surgery is used as the input data, and the assistant who is the role of the person who makes the remarks that assist the surgery is used as the correct output data. The trained model learned in this way outputs the role of the speaker, for example, the surgeon, assistant, attending physician, nurse, speaker, viewer, manager, worker, etc. when text information is input. do.

When there are a plurality of voices extracted from the sound data, the estimation module 2035 estimates the role of each voice from the text information in which the content of the voice is converted. The estimation module 2035 stores the estimated role in the text information database 2021 together with the text information.

After the role is estimated once, the estimation module 2035 assigns the same role to the same voice and the presumable voice, and does not have to execute the process of estimating the role again.

The estimation module 2035 may redo the estimation of the role at a predetermined timing. The predetermined timing is, for example, as follows.
・ Elapsed preset time ・ Switching recordings ・ Appearance of new people

The presentation module 2036 presents the text data stored in the text information database 2021 to the user in response to the request from the user.

<2 data structure>
FIG. 3 is a diagram showing the data structures of the text information database 2021 and the voice information database 2022 stored in the server 20.

As shown in FIG. 3, the text information database 2021 includes an item "date and time", an item "text ID", an item "voice ID", an item "data", and the like.

The item "date and time" is information indicating the date and time when the sound that is the source of the text data is collected.

The item "text ID" indicates information for identifying text data.

The item "voice ID" indicates information for identifying the sound data that is the source of the text data. For example, the text ID "T001" indicates that it was generated based on the voice ID "V001".

The item "data" stores text data. The text data stored in the item "data" includes the text information in which the content of the voice is converted and the role estimated from the text information.

As shown in FIG. 3, the voice information database 2022 includes an item "date and time", an item "voice ID", an item "data", and the like.

The item "date and time" is information indicating the date and time when the sound was collected.

The item "voice ID" indicates information for identifying the acquired sound data.

The item "data" stores sound data. The sound data stored in the item "data" is stored in a data format such as wav.

<3 Summary>
FIG. 4 is a diagram showing an outline of the system 1. In the example shown in FIG. 4, the sound collecting device 40 is installed around the person A and the person B whose voice is to be acquired.

The sound collecting device 40 acquires the sound around the sound collecting device 40. The sound collector 40 transmits a sound signal for the acquired sound to the edge server 30.

The edge server 30 transmits the received sound signal to the server 20.

The server 20 analyzes the sound data of the received sound signal and extracts the sound from the sound data. The server 20 converts the content of the voice into text information by executing the voice recognition process on the extracted voice. The server 20 estimates the role of the voice speaker from the converted text information.

As a result, the server 20 can store the content of the voice uttered by the utterer in association with the role of the utterer and the text information.

<4 operation>
Hereinafter, a series of processes when the server 20 generates text data based on the sound collected by the sound collecting device 40 will be described.

FIG. 5 is a flowchart showing a series of processes when the control unit 203 of the server 20 generates text data based on the sound data. In the following description, for example, a case where a person A and a person B are around the sound collecting device 40 will be described as an example.

The sound collecting device 40 collects ambient sounds. At this time, for example, it is assumed that the person A makes a predetermined remark and then the person B responds to the remark of the person A. The sound collected by the sound collecting device 40 includes the voice of the person A followed by the voice of the person B. The sound collecting device 40 transmits a sound signal about the collected sound to the server 20 via the edge server 30.

In step S501, the control unit 203 acquires sound data from the sound signal received from the edge server 30.

In step S502, the control unit 203 analyzes the acquired sound data. Specifically, for example, the control unit 203 is selected from a group consisting of voice features included in the acquired sound data, for example, voice volume, pitch, voiced, unvoiced, phoneme type, formant, and the like. Analyze at least one. The control unit 203 extracts the voice generated by the person A from the sound data as the first voice having the first feature. The control unit 203 extracts the voice generated by the person B after the person A from the sound data as the second voice having the second feature.

Here, a case where the control unit 203 extracts voice from sound data based on the characteristics of voice has been described as an example. The control unit 203 utilizes at least one method selected from a group consisting of voice characteristics, the direction of the sound collector 40, the timing at which the sound is collected, the sound collector used for the sound collection, and the like. Sound may be extracted.

In step S503, the control unit 203 converts the content of the voice into text information by executing the voice recognition process on the extracted voice. Specifically, for example, the control unit 203 converts the content of the first voice into the first text information by executing the voice recognition process for the first voice. The control unit 203 stores the first text information in the text information database 2021. Further, the control unit 203 converts the content of the second voice into text information by executing the voice recognition process for the second voice. The control unit 203 stores the second text information in the text information database 2021.

In step S504, the control unit 203 estimates the role of the voice speaker based on the text information. Specifically, for example, the control unit 203 inputs the first text information to the trained model. The trained model outputs the first role when the first text information is input. Further, the control unit 203 inputs the second text information to the trained model. The trained model outputs the second role when the second text information is input. The control unit 203 associates the first text information with the first role, associates the second text information with the second role to obtain text data, and stores the text data in the text information database 2021.

In step S505, the control unit 203 presents the text data stored in the text information database 2021 to the user in response to the request from the user.

<5 screen example>
6 to 8 are diagrams showing a display example of a terminal operated by the user when presenting text data to the user in the first embodiment. The user terminal may be, for example, a stationary PC (Personal Computer) or a laptop PC. The user terminal may also function as a head-mounted display, for example, as a transmissive, non-transparent, or see-through head-mounted display. The text data is not limited to the display on the display, and may be printed out on paper and presented to the user.

FIG. 6 is a diagram showing an example of displaying text data when the person A is a surgeon and the person B is an assistant.

In FIG. 6, the objects 601 and 607 represent roles estimated based on the first text information. In FIG. 6, the objects 601 and 607 are located at the left edge of the screen and are displayed as "surgeon". The object 604 represents a role estimated based on the second text information. In FIG. 6, the object 604 is located at the right edge of the screen and is displayed as "assistant". In this way, by changing the position where the object is displayed according to the role, the user can grasp that a person having a different role is talking only by visually recognizing the display position of the role.

FIG. 6 shows an example in which the objects 601 and 607 are located along the left edge of the screen and the object 604 is located along the right edge of the screen, but the positions of the objects 601 and 607 and the object 604 are not limited to this. .. Objects 601 and 604 may be located at the same end.

The icons 602, 608 and 605 represent icons according to their roles. For example, the icons 602 and 608 are displayed below the objects 601 and 607, respectively, to indicate an icon that identifies the surgeon. The icon 605 is displayed below the object 604 and indicates an icon for identifying the assistant. The icon may be automatically set by the control unit 203 according to the role, for example.

In the boxes 603, 609 and 606, text information representing the content of each speaker's remark is displayed. For example, the boxes 603 and 609 are displayed near the right edge of the screen and display the surgeon's remarks together with the time. Further, the box 606 is displayed near the left edge of the screen, and displays the assistant's remarks together with the time.

As a result, the user can confirm the contents of the conversation between the surgeon and the assistant during the operation or the like as text information in a form that identifies each role. For this reason, it is possible to utilize the intraoperative conversation between the surgeon and the assistant, for example, to confirm whether the instructions are appropriate and whether the judgment is incorrect when instructing the trainee. Become. In addition, the surgeon himself can check the conversation during the surgery that he was in charge of, which can be useful for looking back on the points of reflection and finding points for improvement.

FIG. 7 is a diagram showing an example of displaying text data when the person A is a speaker and the person B is a viewer.

In FIG. 7, the objects 701 and 707 represent roles estimated based on the first text information, similar to the objects 601 and 607 in FIG. In FIG. 7, the objects 701 and 707 are located at the left edge of the screen and are displayed as "speaker". The object 704, like the object 604 in FIG. 6, represents a role estimated based on the second text information. In FIG. 7, the object 704 is located at the right edge of the screen and is displayed as "viewer".

The icons 702, 708 and the icon 705 represent role-based icons, similar to the icons 602, 608 and 605 in FIG. For example, the icons 702 and 708 are displayed below the objects 701 and 707, respectively, to indicate an icon that identifies the speaker. The icon 705 is displayed below the object 704 and indicates an icon that identifies the viewer.

Similar to the boxes 603, 609 and 606 in FIG. 6, the boxes 703, 709 and 706 display text information representing the content of each speaker's remark. For example, the boxes 703 and 709 are displayed near the right edge of the screen and display the speaker's remarks together with the time. Further, the box 706 is displayed near the left edge of the screen, and displays the viewer's remarks together with the time.

As a result, the user can confirm the content of the conversation between the speaker and the viewer during the lecture, for example, the question and answer session, as text information in a form that identifies each role. Therefore, the lecturer can confirm the flow of the talk in the lecture, the reaction of the viewer, etc. by confirming the contents of the question and answer as text information. In addition, the user who creates the minutes can easily create the minutes by confirming the contents of the question and answer as text information.

FIG. 8 is a diagram showing an example of displaying text data when the person A is an administrator and the person B is a worker.

In FIG. 8, the objects 801 and 807 represent roles estimated based on the first text information, similar to the objects 701 and 707 in FIG. 7. In FIG. 8, the objects 801 and 807 are located at the left edge of the screen and are displayed as "administrator". The object 804, like the object 704 in FIG. 7, represents a role estimated based on the second text information. In FIG. 8, the object 804 is located at the right edge of the screen and is displayed as "worker".

The icons 802, 808 and the icon 805 represent role-based icons, similar to the icons 702, 708, and 705 in FIG. 7. For example, the icons 802 and 808 are displayed below the objects 801 and 807, respectively, to indicate an icon that identifies the administrator. The icon 805 is displayed below the object 804 and indicates an icon that identifies a worker.

Similar to the boxes 703, 709 and 706 in FIG. 7, the boxes 803, 809 and 806 display text information representing the content of each speaker's remark. For example, the boxes 803 and 809 are displayed near the right edge of the screen, and display the remarks of the administrator together with the time. Further, the box 806 is displayed near the left edge of the screen, and displays the worker's remarks together with the time.

As a result, the user can confirm the contents of the conversation between the manager and the worker at the work site, for example, the work instruction of the day, as text information in the form of identifying each role. As a result, the administrator can check the contents of the work instruction as text information, which can be useful for looking back on the work contents of the day and for planning the work plan for the next day. In addition, the supervisor who manages and supervises the manager can confirm the contents of instructions given by the manager to the worker, the reaction of the worker, and the like as text information. Therefore, when a problem such as harassment occurs, the supervisor can confirm whether the instruction method is appropriate and whether the worker is forced to bear an unreasonable burden.

In this way, the server 20 extracts the voice from the sound data, converts the extracted voice into text information, and estimates the role of the speaker based on the converted text information. Further, the server 20 estimates the roles of the plurality of vocalists from the received one or a plurality of sound data. Therefore, the server 20 can present the text information to the user while determining the role of the speaker even if there is no information registered in advance about the speaker.

<6 Modification example>
In the above embodiment, the case where the voice analysis is performed on the server 20 has been described, but the voice analysis may be performed on a server other than the server 20. For example, the edge server 30 may perform voice analysis and transmit text information to the server 20. Further, the sound collecting device 40 may perform voice analysis and transmit the text information to the edge server 30. Even when the text information is transmitted to the server 20, the sound signal may be transmitted to the server 20.

Further, in the above embodiment, the case where the estimation process is performed on the server 20 has been described, but the estimation process may be performed on a server other than the server 20 as long as it is after the voice analysis. For example, when the edge server 30 or the sound collecting device 40 performs voice analysis, the edge server 30 may perform estimation processing and transmit information on the role to the server 20. Further, when the sound collecting device 40 performs voice analysis, the sound collecting device 40 may perform estimation processing and transmit information regarding the role to the edge server 30.

<Second embodiment>
In the first embodiment, the case where only the sound collecting device 40 is used has been described. However, the method of extracting voice is not limited to this. In the second embodiment, a method of using the photographing device 50 in addition to the sound collecting device 40 will be described. It should be noted that the detailed description of those having the same reference numerals as those of the first embodiment will not be repeated.

<1 Configuration diagram of the entire system>
FIG. 9 is a diagram showing the overall configuration of the system 1A in the second embodiment.

As shown in FIG. 9, the system 1A includes a server 20A, an edge server 30, a sound collecting device 40, and a photographing device 50. The server 20A and the edge server 30 communicate with each other via the network 80. The edge server 30 is connected to the sound collecting device 40 and the photographing device 50. For example, the sound collecting device 40 and the photographing device 50 are transmission / reception devices based on a communication standard used in a short-range communication system between information devices. Specifically, the sound collecting device 40 and the photographing device 50 use a 2.4 GHz band such as a Bluetooth (registered trademark) module, and a beacon signal from another information device equipped with the Bluetooth (registered trademark) module. To receive. The edge server 30 acquires information transmitted from the sound collecting device 40 and the photographing device 50 based on the beacon signal using the short-range communication. In this way, the sound collecting device 40 and the photographing device 50 transmit the acquired voice information of the speaker and the operation information of the speaker to the edge server 30 by short-range communication without going through the network 80. The edge server 30 may communicate with the sound collecting device 40 and the photographing device 50 via the network 80.

The photographing device 50 is a device for receiving light by a light receiving element and outputting it as a photographed image. The photographing device 50 photographs an image in a set direction, and transmits an image signal based on the image data obtained by the imaging to the edge server 30. As the photographing device 50, for example, any of the following devices is assumed.
・ Visible light camera ・ Infrared camera ・ Ultrasonic camera ・ Ultrasonic sensor ・ RGB-D camera ・ LiDAR (Light Detection and Ranging)
Although FIG. 9 shows an example in which the number of photographing devices 50 is one, a plurality of photographing devices 50 accommodated in the system 1A may be provided.

The edge server 30 receives the sound signal transmitted from the sound collecting device 40, and transmits the received sound signal to the server 20. Further, the edge server 30 receives the image signal transmitted from the photographing device 50, and transmits the received image signal to the server 20.

<1.1 Configuration of server 20A>
FIG. 10 is a diagram showing a functional configuration of the server 20A in the second embodiment.

The acquisition module 2033A acquires sound data from the sound signal received by the reception control module 2031. The acquisition module 2033A stores the acquired sound data in the voice information database 2022. The acquisition module 2033A acquires image data from the image signal received by the reception control module 2031. The acquisition module 2033A stores the acquired image data in the image information database 2023. The acquisition module 2022A stores, for example, the acquired sound data and image data in the audio information database 2022 and the image information database 2023, respectively, when the predetermined requirements are satisfied. The predetermined requirements are, for example,:
-From the input of the recording / recording start instruction to the input of the recording / recording end instruction-Achievement of a preset time-Continuous generation of sound (for example, when sound is generated, recording / recording is started, Recording / stop recording if no sound is generated for a preset period)
-Detects the movement of the speaker (for example, recording / recording is started when the movement of the mouth of the speaker is detected, and recording / recording is stopped when the movement does not occur for a preset period).
-Detects an action in which a speaker specifies another speaker (for example, when a speaker who was recording and shooting detects an action in which a different speaker is specified, recording and shooting of the specified target is started, and the recording and shooting of the specified target are started. Recording and shooting will be stopped if the operation does not occur for a preset period.)

The image information database 2023 stores image data based on an image taken by the image pickup device 50 by the server 20A.

The image analysis module 2037 analyzes the acquired image data to extract operation information from the image data. For example, the image analysis module 2037 uses the trained model and extracts motion information from the image captured by the imaging device 50.

In the present embodiment, the trained model is trained to output motion information with respect to the acquired image data, for example. At this time, for the learning data, for example, an image including a predetermined motion is used as input data, and labeling to the motion target and displacement of the labeled target are used as correct output data. For example, an image including a person is used as input data, and labeling of the person's mouth and displacement of the labeled mouth are used as correct output data. The labeling of the limbs of a person and the displacement of the labeled limbs may be used as correct output data.

The image analysis module 2037, for example, extracts motion information of a person's mouth taken from the acquired image data. The extracted motion information is not limited to the mouth, and may be a gesture or the like. The image analysis module 2037 transmits the extracted operation information to the voice analysis module 2034A.

The voice analysis module 2034A extracts voice from the acquired sound data and the operation information obtained by the image analysis. Specifically, the voice analysis module 2034A recognizes, for example, the voice uttered in synchronization with the motion information as the voice of the person, and extracts it from the sound data as the voice of the person. More specifically, the voice uttered in synchronization with the movement of the mouth is recognized as the utterance of the person whose mouth has moved, and is used as the voice of that person.

The voice analysis module 2034A may extract the voice of each person from the sound data when a plurality of people are included in the shooting direction. Further, the voice analysis module 2034A may extract the voice based on the characteristics of the voice, the direction in which the sound is collected, the timing at which the sound is collected, and the sound collecting device that collects the sound. The voice analysis module 2034A may extract the voice of the speaker alone, or may extract the voice of the speaker by combining a plurality of methods.

<2 data structure>
FIG. 11 is a diagram showing a data structure of the image information database 2023 stored in the server 20A.

As shown in FIG. 11, the image information database 2023 includes an item "date and time", an item "image ID", an item "voice ID", an item "data", and the like.

The item "date and time" is information indicating the date and time when the image was recorded.

The item "image ID" indicates information for identifying image data.

The item "voice ID" indicates information for identifying the associated sound data. The image data and the sound data are associated with each other based on, for example, time information.

The item "data" stores image data. The image data stored in the item "data" is stored in a data format such as jpg.

<3 Summary>
FIG. 12 is a diagram showing an outline of the system 1A in the second embodiment. In the example shown in FIG. 12, the sound collecting device 40 is installed around the person A and the person B whose voice is to be acquired. Further, the photographing device 50 is installed so as to include the person A and the person B in the photographing direction.

The sound collecting device 40 acquires the sound around the sound collecting device 40. The sound collector 40 transmits the acquired sound signal to the edge server 30.

The photographing device 50 acquires an image in the photographing direction. The photographing device 50 transmits the acquired image signal to the edge server 30.

The edge server 30 transmits the received sound signal and the image signal to the server 20A.

The server 20A refers to the image analysis result of the image data, and extracts the sound of the person being photographed from the sound data of the received sound signal. The server 20A converts the content of the voice into text information by executing the voice recognition process on the extracted voice. The server 20A estimates the role of the voice speaker from the converted text information.

As a result, the server 20A can store the content of the voice uttered by the utterer in association with the role of the utterer and the text information.

As a result, the server 20A can more accurately extract voice from the acquired sound data and image data and convert it into text information. Therefore, the server 20A can accurately extract the voice of the speaker even when the voice of the speaker is small and it is difficult to differentiate from the surrounding sound.

<4 operation>
Hereinafter, a series of processes when the server 20A generates text data based on the sound collected by the sound collecting device 40 and the operation photographed by the photographing device 50 will be described.

FIG. 13 is a flowchart showing a series of processes when the control unit 203A of the server 20A generates text data based on the sound data and the image data. In the following description, for example, there are a person A and a person B around the sound collecting device 40 as shown in FIG. 12, and the photographing device 50 is installed so as to include the person A and the person B in the photographing direction. Let's take an example.

The sound collecting device 40 collects ambient sounds. At this time, for example, it is assumed that the person A makes a predetermined remark and then the person B responds to the remark of the person A. The sound collected by the sound collecting device 40 includes the voice of the person A followed by the voice of the person B. The sound collecting device 40 transmits a sound signal about the collected sound to the server 20A via the edge server 30.

The photographing device 50 acquires an image in the photographing direction. The image captured by the photographing apparatus 50 includes the motion of the person A and the motion of the person B. The photographing device 50 transmits an image signal about the captured image to the server 20A via the edge server 30.

In step S1301, the control unit 203A acquires image data from the image signal received from the edge server 30.

In step S1302, the control unit 203A extracts operation information from the image data by analyzing the acquired image data. The control unit 203A extracts, for example, motion information about the motions of the person A and the person B included in the shooting direction, for example, the movement of the mouth accompanying the speech, the gesture, and the like.

In step S1303, the control unit 203A analyzes the sound data based on the image analysis result of the acquired image data. Specifically, the control unit 203A recognizes the voices uttered in synchronization with the movements of the mouths of the person A and the person B as the voices of the person A and the person B, and as the voices of the person A and the person B. Extract from sound data.

Here, the case where the control unit 203A extracts the voice from the sound data based on the characteristics of the voice and the motion information of the speaker, particularly the movement of the mouth, has been described as an example. The control unit 203A includes at least one selected from the group consisting of voice characteristics, the direction of the sound collecting device 40, the timing at which the sound is collected, the sound collecting device used for sound collecting, and the like, and the photographing device 50. The voice may be extracted by using in combination with at least one selected from the group consisting of other motions of the vocalist, for example, gestures accompanying the vocalization, movements specifying different vocalizers, and the like.

<5 Modification example>
In the above embodiment, the case where the image analysis and the voice analysis based on the image analysis result are performed on the server 20A has been described, but a series of analysis processes may be performed on a server 20 other than the server 20. For example, the edge server 30 may perform image analysis and voice analysis based on the image analysis result, and transmit text information to the server 20A. Further, the photographing device 50 may perform image analysis and transmit the result of the image analysis to the sound collecting device 40, whereby the sound collecting device 40 may perform voice analysis and transmit the text information to the edge server 30. ..

Further, in the above embodiment, the case where the estimation process is performed on the server 20A has been described, but the estimation process may be performed on a server other than the server 20A as long as it is after the voice analysis. For example, when the edge server 30 or the sound collecting device 40 performs voice analysis, the edge server 30 may perform estimation processing and transmit information on the role to the server 20A. Further, when the sound collecting device 40 performs voice analysis, the sound collecting device 40 may perform estimation processing and transmit information regarding the role to the edge server 30.

Further, in the above embodiment, the case where the estimation module 2035 estimates the role of the speaker using the trained model has been described as an example. However, the estimation module 2035 may estimate the role of the speaker without using the trained model. For example, the storage unit 202 stores in advance a table in which a role and a predetermined wording are associated with each other. The estimation module 2035 refers to the table and estimates the role from the text information.

<Additional notes>
The matters described in each of the above embodiments are added below.

(Appendix 1)
A program for causing a computer 20 including a processor 29 and a memory 25 to execute the program, the program has a step (S501) of acquiring the sound collected by the sound collecting device 40 in the processor 29, and the acquired sound. From the above, the step of extracting at least one sound (S502), the step of converting the extracted sound into text information (S503), and the role of the speaker of the extracted sound are estimated based on the text information. A program for executing a step (S504) and a step (S505) of presenting the converted text information to the user so that the role can be identified.

(Appendix 2)
The program according to Supplementary Note 1, wherein at least one voice is extracted based on the information regarding the characteristics of the voice in the extraction step (S502). (Paragraph 0063)

(Appendix 3)
The program according to Supplementary Note 1, wherein at least one voice is extracted based on the information regarding the direction of the sound in the extraction step (S502). (Paragraph 0063)

(Appendix 4)
The program according to Appendix 1, which extracts at least one voice based on the information regarding the timing of acquiring the sound in the extraction step (S502). (Paragraph 0063)

(Appendix 5)
In the step (S502) of causing the processor 29 to execute and extract the step (S1301) of acquiring the image captured by the photographing device and the step (S1302) of acquiring the motion information of the speaker from the acquired image. The program according to Appendix 1, which extracts a voice based on the timing at which the sound is collected and the timing at which the operation information is acquired. (Paragraph 0995)

(Appendix 6)
The program according to Appendix 5, wherein the motion information is motion information of the mouth or limbs of the speaker taken by the photographing device 50. (Paragraph 0995)

(Appendix 7)
The program according to any one of Supplementary Provisions 1 to 6, which estimates the role of the speaker based on the information of the preset role in the estimation step (S504). (Paragraph 0039)

(Appendix 8)
In the estimation step (S504), the character information about a predetermined remark is used as input data, and the role of the speaker is used as the correct answer output data. The program according to any one of Supplementary Provisions 1 to 6 for estimating.
(Paragraph 0040)

(Appendix 9)
In the step (S502) of extracting and converting a plurality of voices in the extraction step (S502), in the step (S504) of converting and estimating a plurality of text information by analyzing each of the extracted plurality of voices. , The program according to any one of Supplementary note 1 to 8, which estimates the role of the speaker of the extracted voices based on the converted text information. (Paragraph 0036)

(Appendix 10)
The program according to Appendix 9, in which in the estimation step (S504), the person in charge of performing the medical practice and the person in charge of assisting the person in charge of performing the medical treatment are estimated as the roles of the voice speakers of the plurality of voices. (Paragraph 0074)

(Appendix 11)
The program according to Appendix 9, which estimates the main speaker and the viewer who watches the speaker's story as the roles of the speakers of the plurality of voices in the estimation step (S504). (Paragraph 0079)

(Appendix 12)
The program according to Appendix 9, which estimates an administrator and a managed person by the administrator as roles of a plurality of voice utterers in the estimation step (S504). (Paragraph 0084)

(Appendix 13)
A method executed by a computer 20 including a processor 29 and a memory 25, wherein the processor 29 acquires a sound collected by the sound collecting device 40 (S501), and the acquired sound is used as a method. A step of extracting at least one voice (S502), a step of converting the extracted voice into text information (S503), and a step of estimating the role of the speaker of the extracted voice based on the text information. (S504), and a step (S505) of presenting the converted text information to the user so that the role can be identified.

(Appendix 14)
In the information processing device 20 including the control unit 203, the control unit 203 has a step (S501) of acquiring the sound collected by the sound collecting device 40 and a step of extracting at least one sound from the acquired sound. (S502), a step of converting the extracted sound into text information (S503), a step of estimating the role of the speaker of the extracted sound based on the text information (S504), and the converted text. The information processing apparatus 20 for executing the step (S505) of presenting information to the user so that the role can be identified.

(Appendix 15)
A means for acquiring the sound collected by the sound collecting device 40 (S501), a means for extracting at least one voice from the acquired sound (S502), and a means for analyzing the extracted voice to convert it into text information. (S503), a means for estimating the role of the speaker of the extracted voice based on the text information (S504), a means for presenting the converted text information to the user so that the role can be identified (S505). A system equipped with.

20 servers, 22 communication IFs, 23 input / output IFs, 25 memories, 26 storages, 29 processors, 30 edge servers, 40 sound collectors, 50 shooting devices, 80 networks, 201 communication units, 202 control units, 203 communication units, 2021 Text information database, 2022 audio information database, 2023 image information database.

Claims (16)

A program for causing a computer having a processor and a memory to execute the program, wherein the program causes the processor to execute the program.
Steps to acquire the sound collected by the sound collector, and
A step of extracting at least one voice from the acquired sound, and
The step of converting the extracted voice into text information by analyzing it,
A step of estimating the role of the speaker of the extracted voice based on the text information, and
The step of presenting the converted text information to the user together with the estimated role for the speaker, and
To execute ,
A program that estimates information about the position of a speaker as the role of the speaker without specifying the speaker in the estimation step. The program according to claim 1, wherein in the step of presenting, the converted text information is presented to the user so that the role can be identified. In the extraction step
The program of claim 1 or 2, wherein the at least one voice is extracted based on information about voice characteristics. In the extraction step
The program according to claim 1 or 2, wherein the at least one voice is extracted based on the information regarding the direction of the sound. In the extraction step
The program according to claim 1 or 2, wherein the at least one voice is extracted based on the information regarding the timing of acquiring the sound. Steps to acquire the image taken by the shooting device,
The processor is made to execute the step of acquiring the operation information of the speaker from the acquired image.
In the extraction step
The program according to claim 1 or 2, wherein the voice is extracted based on the timing at which the sound is collected and the timing at which the operation information is acquired. The program according to claim 6, wherein the motion information is motion information of the mouth or limbs of the speaker. In the estimation step
The program according to any one of claims 1 to 7, which estimates the role of the speaker based on the preset information of the role. In the estimation step, the character information about a predetermined remark is used as input data, and the role of the speaker is played by inputting the text information into the trained model learned as the correct output data. The program according to any one of claims 1 to 7. In the extraction step, a plurality of voices are extracted and
In the conversion step, the extracted voices are analyzed to convert them into a plurality of text information.
The program according to any one of claims 1 to 9, wherein in the estimation step, the roles of the speaker of the extracted voices are estimated based on the converted text information. In the estimation step
The program according to claim 10, wherein the person in charge of performing the medical practice and the person in charge of assisting the person in charge of performing the medical treatment are estimated as the roles of the voice speakers of the plurality of voices. In the estimation step
The program according to claim 10, wherein the main speaker and the viewer who listens to the talk of the speaker are estimated as the roles of the speaker of the plurality of voices. In the estimation step
The program according to claim 10, wherein the manager and the managed person by the manager are estimated as the roles of the voice speakers of the plurality of voices. A method for causing a computer equipped with a processor and a memory to execute the method, wherein the processor is used.
Steps to acquire the sound collected by the sound collector, and
A step of extracting at least one voice from the acquired sound, and
The step of converting the extracted voice into text information by analyzing it,
A step of estimating the role of the speaker of the extracted voice based on the text information, and
The step of presenting the converted text information to the user together with the estimated role for the speaker, and
And run
A method of estimating information about the position of the speaker as the role of the speaker without specifying the speaker in the estimation step. An information processing device including a control unit, wherein the control unit
Steps to acquire the sound collected by the sound collector, and
A step of extracting at least one voice from the acquired sound, and
The step of converting the extracted voice into text information by analyzing it,
A step of estimating the role of the speaker of the extracted voice based on the text information, and
The step of presenting the converted text information to the user together with the estimated role for the speaker, and
And run
An information processing device that estimates information about the position of the speaker as the role of the speaker without specifying the speaker in the estimation step. A means of acquiring the sound collected by the sound collector,
A means for extracting at least one voice from the acquired sound, and
A means of converting the extracted voice into text information by analyzing it,
A means for estimating the role of the speaker of the extracted voice based on the text information, and
A means of presenting the converted text information to the user together with an estimated role for the speaker.
Equipped with
In the estimation means, a system that estimates information about the position of the speaker as the role of the speaker without specifying the speaker.

Images