JP2021156764A - Calibration equipment, mobile identification equipment used in the calibration equipment, training equipment for it, and computer programs for them. - Google Patents

Abstract

PROBLEM TO BE SOLVED: To automatically calibrate a different type of sensor.
SOLUTION: A calibration device receives first and second time-series data for acquiring first and second time-series data regarding the positions of a predetermined number of moving objects by two sensors, and receives the first and second time-series data. First and second moving bodies represented by the second time-series data For each combination with the second moving body, the time-series data of the positions of the first and second moving bodies are received, and the first and second moving bodies are received. The correspondence between the moving body identification device trained to output the score indicating whether or not the moving bodies are the same moving body and the corresponding relationship of each moving body represented by the first and second time series data based on the output. A sensor that estimates and calibrates the position and orientation of the second sensor with respect to the first sensor so that the output errors of the first and second sensors for each moving object satisfy certain conditions using the correspondence. Including calibrator.
[Selection diagram] Fig. 5

Description

The present invention relates to a technique for calibrating a moving body sensor, and more particularly to a technique for identifying a moving body such as a plurality of persons detected by a plurality of moving body sensors for calibration of the moving body sensor.

RGB-D sensors and microphone arrays are widely used to represent the visual and auditory environment in a machine-understandable manner. A plurality of these sensors are required to express the expanse of the environment. When using a plurality of sensors, information on the position and orientation of each sensor is required in order to share the outputs of those sensors and properly combine them with each other. However, manually calibrating various types of sensors for this purpose is cumbersome and time consuming. Therefore, a technique capable of calibrating a plurality of sensors without human intervention is desirable.

Non-Patent Document 1 described later proposes a skeleton-based viewpoint invariant conversion in order to guide the conversion from the position information of the human body to the position and posture information of the RGB-D sensor. In this conversion, the relative positions and orientations of these two sensors are calculated using the common human body (skeleton) observed by the two adjacent sensors.

Non-Patent Document 2 described later proposes an algorithm for calibrating an RGB-D sensor and automatically recalibrating it using information on the observed joint positions of the skeleton.

On the other hand, microphone arrays are widely used to perceive the auditory environment and improve the hearing of robots. However, most of the techniques for perceiving the environment using microphone arrays are manual calibration, and only a few techniques for automatically calibrating multiple microphone arrays. Was being done.

Y. Han, S.-L. Chung, J.-S. Yeh, and Q.-J. Chen, “Localization of rgb-d camera skeleton-based viewpoint invariance transformation,” vol. 63, 10 2013, pp. 1525-1530 K. Desai, B. Prabhakaran, and S. Raghuraman, “Skeleton-based continuous extrinsic calibration of multiple rgb-d kinect cameras,” in Proceedings of the 9th ACM Multimedia Systems Conference, ser. MMSys '18. New York, NY, USA: ACM, 2018, pp. 250-257. [Online]. Available: http://doi.acm.org/10.1145/3204949.3204969 J. Gilmer, SS Schoenholz, PF Riley, O. Vinyals, and GE Dahl, “Neural message passing for quantum chemistry,” in Proceedings of the 34th International Conference on Machine Learning --Volume 70, ser. ICML'17. JMLR.org , 2017, pp. 1263-1272. S. Agarwal, N. Snavely, SM Seitz, and R. Szeliski, “Bundle adjustment in the large,” in Proceedings of the 11th European Conference on Computer Vision: Part II, ser. ECCV'10. Berlin, Heidelberg: Springer- Verlag, 2010, pp. 29-42.

As mentioned above, the technique of automatically calibrating a plurality of sensors has been limited to the technique among sensors of the same type. There is no technique for automatically calibrating different types of sensors, such as RGB-D sensor sensors and microphone arrays.

Therefore, the present invention provides a calibrator that automatically calibrates different types of sensors, a mobile identification device used in the calibrator, a training device for that purpose, and a computer program for them.

The calibrator according to the first aspect of the present invention calibrates the positions and orientations of the first sensor and the second sensor, each capable of detecting and outputting the positions of a plurality of moving objects in a discrete time series. A calibrator for this purpose, the first time-series data and the second time-series data regarding the positions of a predetermined number of moving objects measured by the first sensor and the second sensor over a predetermined time, respectively. With the acquisition unit to be acquired and the first time-series data and the second time-series data as inputs, the first moving body represented by the first time-series data and the second time-series data represented by the second time-series data. For each combination with the two moving bodies, the time-series data of the position of the first moving body in the first time-series data and the time-series data of the position of the second moving body in the second time-series data. As input, a moving body consisting of a neural network pre-trained to output a score indicating whether the first moving body and the second moving body forming the combination are the same moving body. Based on the output of the identification means and the moving body identification means, the correspondence between each moving body represented by the first time-series data and each moving body represented by the second time-series data is estimated, and the corresponding relationship is estimated. Sensor calibration that calibrates the position and orientation of the second sensor with respect to the first sensor so that the output error between the first sensor and the second sensor for each moving object satisfies a predetermined condition using a correspondence relationship. Including means.

Preferably, the sensor calibration means includes a minimization means that calibrates the position and orientation of the second sensor so as to minimize the sum of the output errors.

More preferably, the calibrator further includes a parallel training means that uses the first time series data and the second time series data to train the moving body identification means in parallel with the operation of the correspondence estimation means. ..

More preferably, the parallel training means inputs the moving body identification means, the output of the moving body identification means, and the position data of each of the first time series data and the second time series data at the same time step. Parameters of the decoder, the moving object identification means and the decoder so that the output of the decoder is close to the position data of the same time step input to the decoder over a predetermined range of the first time series data and the second time series data. Includes coordinating means for training mobile identification means by coordinating.

Preferably, the adjusting means uses the first time series data and the second time series data over the entire predetermined time, and uses the error between the output of the decoder and the position data of the same time step input to the decoder. It includes an error back-propagation means for training the moving body identification means by adjusting the parameters of the moving body identification means and the decoder by the error back-propagation method.

More preferably, the parallel training means trains the mobile identification means each time the first time series data and the second time series data are given.

The mobile body identification device according to the second aspect of the present invention is a first moving body and a second moving body regarding the positions of the first moving body and the second moving body, respectively, measured by the first sensor and the second sensor over a predetermined time. Pre-trained neural to output a score indicating whether the first moving body and the second moving body are the same moving body by inputting the time series data and the second time series data of・ Consists of a network.

Preferably, each of the first time-series data and the second time-series data includes the position data of the target moving object at predetermined time intervals, and each of the position data at predetermined time intervals is the target moving object. Includes the position and speed of the data and time information indicating the time when the position and speed were measured.

More preferably, the neural network has a plurality of inputs that receive the position and velocity contained in the first time series data and the position and velocity contained in the second time series data, and an output that outputs the probability. It is a neural network consisting of multiple layers.

The training device according to the third aspect of the present invention includes a time series data acquisition unit that acquires a time series of position data obtained in a predetermined time step for each of a plurality of moving objects over a predetermined time, and a time series data acquisition unit. Position data extraction means that extracts position data acquired at the same time in a specified order from the time series of position data acquired by the series data acquisition unit, input determined by the number of predetermined time steps, and at least A first neural network with one output, a second neural network with the same number of inputs and outputs, both determined by the position data constituting the time series, and two moving objects from multiple moving objects. Of the time series of position data, the time series of the position data of the moving body constituting the extracted combination is used as the input to the first neural network, and the first neural network is used. The input means given to, the first sampling means that samples the values output by the first neural network in response to the input, and the values sampled by the first sampling means for each of the possible combinations. Among the selection means for selecting the combination in which the largest value is obtained and the position data extracted by the position data extraction means, the position data of the two moving bodies corresponding to the combination selected by the selection means are selected. A second sampling means that inputs to the second neural network and samples the output of the second neural network, two moving body position data given to the input of the second neural network, and a second. By the error backpropagation method, the parameters of each of the first neural network and the second neural network are reduced so that the error between the sampling means of the second neural network and the sampled value is small. A parameter adjusting means for adjusting, a position data extracting means, a first neural network, an input means, a first sampling means, a selection means, a second sampling means, and a parameter adjusting means are used in a time series of position data. The first repeat execution means that specifies the position data in order from the beginning and operates repeatedly until the time series data ends, and the first repeat execution means repeat until a predetermined end condition is satisfied. The parameters of the second repeat execution means to be executed and the first neural network at the time when the repetition by the second repeat execution means is completed are determined. Includes parameter storage means for storage in a fixed storage device.

Preferably, the parameter adjusting means is between the two moving body position data given to the input of the second neural network and the values sampled by the second sampling means from the output of the second neural network. Backpropagation of errors so that the error accumulated by the error accumulating means becomes smaller after the error accumulating means for accumulating the error a predetermined number of times and the first sampling means and the second sampling means operate a predetermined number of times. It includes a batch adjustment means for adjusting each parameter of the first neural network and the second neural network by batch processing according to the method.

The computer program according to the fourth aspect of the present invention causes the computer to function as any of the above-mentioned calibration devices.

The computer program according to the fifth aspect of the present invention causes the computer to function as any of the above-mentioned mobile identification devices.

The computer program according to the sixth aspect of the present invention causes the computer to function as any of the training devices described above.

The above and other objectives, features, aspects and advantages of the invention will become apparent from the following detailed description of the invention as understood in connection with the accompanying drawings.

FIG. 1 is a diagram schematically showing a configuration of a sensor system. FIG. 2 is a schematic diagram showing a calibration method between two sensors. FIG. 3 is a diagram schematically showing a factor graph showing the calibration process. FIG. 4 is a diagram schematically showing a factor graph having edges only between the same moving bodies in the calibration process. FIG. 5 is a block diagram showing an outline of the overall configuration of the automatic sensor calibration system according to the embodiment of the present invention. FIG. 6 is a diagram showing the appearance of a computer that realizes the automatic calibration system shown in FIG. FIG. 7 is a block diagram showing a hardware configuration of a computer whose appearance is shown in FIG. FIG. 8 is a diagram schematically showing a schematic configuration of the autoencoder shown in FIG. FIG. 9 is a diagram schematically showing a configuration of a neural network for transmitting a message disclosed in Non-Patent Document 3. FIG. 10 is a diagram schematically showing a configuration of an encoder which is a part of an autoencoder used in the embodiment of the present invention, which is inspired by the neural network shown in FIG. FIG. 11 is a flowchart showing a schematic control structure of a computer program for operating the computer shown in FIGS. 6 and 7 as a training device for training the encoder shown in FIG. FIG. 12 outlines the control structure of a computer program for operating the computer shown in FIGS. 6 and 7 as a calibrator that calibrates a plurality of sensors using an encoder trained by the program shown in FIG. It is a flowchart. FIG. 13 is a flowchart showing a schematic control structure of a computer program for operating the computer shown in FIGS. 6 and 7 as an online training device for training the encoder shown in FIG. 10 online during its operation. FIG. 14 is a schematic control structure of a computer program for operating the computer shown in FIGS. 6 and 7 as a training device according to a second embodiment of the present invention for training the encoder shown in FIG. It is a flowchart which shows.

In the following description and drawings, the same parts are given the same reference numbers. Therefore, detailed explanations about them will not be repeated. In the following embodiments, in order to facilitate understanding, a case where the number of sensors to be calibrated is two or three at the same time will be described unless otherwise specified. However, the present invention is not limited to such an embodiment, and can be realized in the same manner as described below when the number of sensors to be calibrated is four or more. In the following explanation, a human being is assumed as a moving object (moving body), but the present invention is not necessarily limited to human beings.

1. 1. 1st Embodiment 1 Configuration (1) Background FIG. 1 shows a schematic configuration of a sensor system 50 using the calibration device according to the 1st embodiment of the present invention. With reference to FIG. 1, the sensor system 50 includes two RGB-D sensors 60 and 62 and one microphone array 64 to detect the positions of two subjects 66 and 68 and time. Output the position data of the series. The RGB-D sensor 60 and the RGB-D sensor 62 are sensors capable of measuring the RGB image and the distance from the sensor to the target, and the three-dimensional coordinates of the target persons 66 and 68 detected by each sensor are set to the position of each sensor. Is output in the local coordinates of each sensor whose origin is. Further, the microphone array 64 is a two-dimensional sensor, and outputs the two-dimensional coordinates of the target persons 66 and 68 obtained by the microphone array 64 in local coordinates with the position of the microphone array 64 as the origin. The microphone array can detect the position of a person only when the person speaks. Therefore, in the collection of training data and the collection of calibration data, which will be described later for a sensor system including a microphone array, it is necessary to make some utterance when a person walks around in a predetermined area.

If the position and posture (orientation) of each local sensor are accurately known, the global coordinates of the person detected by each sensor and their correspondence can also be known. Therefore, the audiovisual environment in which these sensors are placed can be easily managed by a computer by combining the outputs of these sensors.

However, when the position or orientation of these sensors is unknown, it is necessary to calibrate the output of each sensor so that the coordinate values output by each sensor can be converted into common global coordinates.

(2) Sensor Calibration With reference to FIG. 2, for example, the local coordinates 80 of the RGB-D sensor 60 are adopted as the global coordinates. In this case, the local coordinates of the RGB-D sensor 62 are translated so that the origin coincides with the origin of the local coordinates 80 so as to be represented by the local coordinates 82. Furthermore each axis of the local coordinate _{82 (e 1 ', e 2} ' and is represented by the unit vector _{e 3} '.) The axes of the local coordinate 80 of RGB-D sensor 60 _(e 1, of _{e 2} and ₃ Rotate the local coordinates to match the unit vector.). T ₂ the coordinate conversion by translational movement at this _time, if indicated coordinate transformation by rotating and R _2, converting to convert the local coordinates of the RGB-D sensor 62 to local coordinates by the local-coordinate 80 of the RGB-D sensor 60 is normal Is expressed by the following formula.

ただしＲ_２は３×３の回転行列、ｔ_２は３×１の平行移動ベクトル、Ｏは３×１のゼロベクトルである。回転行列Ｒ_２は２次元の特殊直交群ＳＯ（２）をなす。

However, R ₂ is a 3 × 3 rotation matrix, t ₂ is a 3 × 1 translation vector, and O is a 3 × 1 zero vector. The rotation matrix R ₂ forms a two-dimensional special orthogonal group SO (2).

For the same person, the three-dimensional sensor output vectors observed by the RGB-D sensors 60 and 62 in m + 1 time steps are p ₁ ⁱ and p ₂ ⁱ (i = 0, 1, ..., M), respectively. Then, the matrix R2 and the vector t2 in the above equation are obtained by the following equations, respectively.

このＲ_２及びｔ_２を記憶しておくことにより、ＲＧＢ−Ｄセンサ６２のローカル座標からＲＧＢ−Ｄセンサ６０のローカル座標、すなわちグローバル座標への変換が行える。この行列Ｒ_２及びｔ_２を求めることがＲＧＢ−Ｄセンサ６０及び６２の校正である。ＲＧＢ−Ｄセンサ６０及び６２はいずれも３Ｄセンサなので、この明細書ではこのような校正を３Ｄ＋３Ｄ校正と呼ぶ。

By _{storing the R 2} and t ₂ , the local coordinates of the RGB-D sensor 62 can be converted into the local coordinates of the RGB-D sensor 60, that is, the global coordinates. Obtaining the matrices R ₂ and t ₂ is the calibration of the RGB-D sensors 60 and 62. Since the RGB-D sensors 60 and 62 are both 3D sensors, such calibration is referred to as 3D + 3D calibration in this specification.

On the other hand, the calibration between the RGB-D sensor 60 and the microphone array 64 can be performed as follows. The microphone array 64 cannot measure the distance to the object, only the direction. Therefore, the calibration of the RGB-D sensor 60 and the microphone array 64 in this way is called 3D + 2D calibration.

The output of the microphone array 64 for a same person to the outputs of _p 3, RGB-D sensor 60 and _{p 1.} This can be expressed as follows.

ただしθ_１はマイクロホン・アレイ６４から対象の人物への方向のアジマス角、θ_２は仰角を表す。

However, θ ₁ represents the azimuth angle in the direction from the microphone array 64 to the target person, and θ ₂ represents the elevation angle.

Conversion from the local coordinates _{p 3} of the microphone array 64 to the local coordinates of the RGB-D sensor 60 (i.e., global coordinates) of the microphone array in the Lie algebra so (3) associated with the above-described special orthogonal group SO (3) It can be easily calculated by the following formula using the matrix ξ representing the posture of 64.

上式の「ｉ」はｉ番目の人物の識別子を、「ｊ」はセンサの識別子を、それぞれ表す。また上式では左辺の「１」を省略してある。ξ_ｊの右上の「＾」は、３行３列の行列ξ_ｊを３×１のベクトルで表していることをす。ｊ＝３の場合、この行列ξ_３は以下の式により求めることができる。

In the above equation, "i" represents the identifier of the i-th person, and "j" represents the identifier of the sensor. In the above equation, the "1" on the left side is omitted. The "^" in the upper right of ξ _j indicates that the 3-by-3 matrix ξ _j is represented by a 3 × 1 vector. When j = 3, this matrix ξ ₃ can be obtained by the following equation.

ここでｍは測定データの先頭を０としたときの最後の測定データの番号であり、ｐ_ｉ ^１はＲＧＢ−Ｄセンサ６０のｉ番目の測定データを表し、ｐ_ｉ ^３はマイクロホン・アレイ６４のｉ番目の測定データを表す。

Where m is the number of the last measurement data when the start of the measurement data and 0, p _i ¹ denotes the i-th measurement data RGB-D sensor 60, p _i ³ is the microphone arrays 64 Represents the i-th measurement data.

_{By obtaining ξ 3} in this way, the measurement data obtained at the local coordinates of the microphone array 64 can be converted into the local coordinates (that is, the global coordinates) of the RGB-D sensor 60.

(3) Factor graph The above calibration is based on the premise that the persons measured by each sensor can be associated with each other. However, in a real environment, due to measurement errors, for example, when the positions of multiple people are measured by multiple sensors, it is difficult to know exactly which person in the output of each sensor corresponds to each other. There's a problem.

Therefore, it is conceivable to estimate a probabilistic expression from the data including such noise. A factor graph is a suitable tool for such estimation problems. Similar to the Bayesian network, the factor graph can express the simultaneous probability as a product of factors.

FIG. 3 shows Graph 100 as an example. In FIG. 3, S ₁ and S ₂ are sensors, x _1i and y _1i (i = 1, 2) are the position data of the first person and the second person measured twice by the _{sensor S 1 , x 2i} and y _2i (i = 1,2) indicates the position data of the first person and second person sensor _{S 2} is measured over the sensor _{S 1} at the same time 2 times. The subscript i indicates a time step. x indicates the position data of the first person, and y indicates the position data of the second person.

These, as shown in FIG. 3, the sensor _{S 1} and _{S 2,} the position data _{x 1i} and _y 1i measured sensor _{S 1} (i = 1,2), as well as position data _{x 2i} and measured sensor _{S 2} Graph 100 consisting of y _2i (i = 1, 2) as a vertex and an edge connecting each sensor and all combinations of the vertices corresponding to the position data measured in the same time step among the position data measured by the sensor. To form.

As shown in FIG. 3, a position sensor S ₁ and S ₂ were measured for the same person, be different from each other in general for measurement error, immediately it can not be associated with one another. In this embodiment, the measurement data are associated with each other using the graph 100 based on the following concept.

That is, referring to FIG. 4, when x ₁₁ and x ₂₁ , y ₁₁ and y ₂₁ , x ₁₂ and x ₂₂ , and y ₁₂ and y ₂₂ represent the same person, the edge 120 connecting them in the graph 100, If the factor graph 100 can be modified so that only 122, 124, and 126 are left and all other edges (represented by dotted lines) are deleted _{, the measurement data of the sensor S 1 and} the measurement data of the sensor S ₂ can be associated with each other. It can be carried out.

In this embodiment, a graph neural network (GNN) is used for this association. GNN is a kind of neural network and is suitable for processing data having a graph structure. Recently, GNN has been found to be very effective in inference and multi-agent interactive systems. In Non-Patent Document 3, GNN is used for local message transmission in a graph for making a pair comparison. The system according to the first embodiment of the present invention described below estimates the match of a person measured by two sensors using GNN, using the description of Non-Patent Document 3 as a hint. The details will be described later.

In the embodiment described below, a neural network for determining whether or not the two time series data belong to the same person is used. The two time-series data are two time-series data consisting of first and second position data output by a plurality of sensors over a predetermined time step. By using this neural network, for example, two time series data output by the first sensor for two people and two time series output by the second sensor for the same two people are compared. , Which time-series data of the first sensor and which time-series data of the second sensor represent the same person are associated with each other. Therefore, using the above description of Non-Patent Document 3 as a hint, the neural network is trained to provide the above-mentioned functions.

(4) Overall Configuration of System FIG. 5 shows the overall configuration of the calibration system 150 according to the first embodiment of the present invention. With reference to FIG. 5, the calibration system 150 includes an autoencoder 178 for training the neural network including the above-mentioned neural network as a part, and parameters of the neural network trained by the autoencoder 178. A parameter storage unit 180 for storing is included. Hereinafter, this neural network will be referred to as an encoder.

The calibration system 150 further includes a training data generation unit 160 for generating training data for training the encoder, a training data collection device 170 for collecting training data generated by the training data generation unit 160, and a training data collection device 170. The training data storage unit 172 for storing the training data collected by the training data collection device 170 in a computer-readable format, and the training program storage unit 176 for storing the computer program for training the above-mentioned encoder. And the program stored in the training program storage unit 176 is executed, and the auto encoder 178 is trained using the training data stored in the training data storage unit 172, whereby the encoder that is a part of the auto encoder 178 is trained. It includes an online calibrator training system 174 for performing training and a parameter storage unit 180 for storing parameters for realizing the auto encoder 178 by a computer after the training is completed.

In this example, the training data generation unit 160 includes RGB-D sensors 60 and 62, and is a time series of a predetermined number of time steps measured for each predetermined time step of the position data of two persons moving in a predetermined area. Suppose to generate. Of course, in training, a number of such training data are generated and stored in the training data storage unit 172.

The calibration system 150 further includes a calibration target acoustic processing system 162 to be calibrated. In this embodiment, the calibration target sound processing system 162 includes RGB-D sensors 60 and 62 like the training data generation unit 160, and acquires a time series of position data of two persons moving in a predetermined area. And. Further, in this example, the purpose is to calibrate the positions and orientations of the RGB-D sensors 60 and 62. Therefore, it is not necessary to strictly set the positions and orientations of the RGB-D sensors 60 and 62 before calibration.

The calibration system 150 further uses the parameters related to the encoder among the parameters stored in the parameter storage unit 180 and the program stored in the auto encoder 176 to realize the algorithm of the encoder from the sound processing system 162 to be calibrated. The online calibration device 182 for processing the obtained time-series position data of a predetermined number of time steps and calibrating between the RGB-D sensor 60 and the RGB-D sensor 62, and the online calibration device 182 for calibration. It includes a calibration parameter storage unit 186 for storing the resulting calibration parameter. In this embodiment, the local coordinates of the RGB-D sensor 60 are set as world coordinates, and the parameters (the above-mentioned matrix R2 and vector t2) for converting the local coordinates of the RGB-D sensor 62 into world coordinates are the online calibration device 182. And stored in the calibration parameter storage unit 186.

The calibration system 150 further updates the background of the calibration device to simultaneously train the encoder using the time-series data obtained during calibration by the online calibration device 182 and during the actual operation of the sound processing system 162 to be calibrated. Includes system 184.

(5) Realization by computer In FIG. 5, each functional unit except the training data generation unit 160 and the audio processing system 162 to be calibrated is realized by computer hardware and a computer program executed on the computer hardware. FIG. 6 shows the appearance of such a computer system 290, and FIG. 7 shows a block diagram of the hardware configuration of the computer system 290.

With reference to FIG. 6, the computer system 290 includes a computer 300 with a DVD drive 310, a keyboard 306, a mouse 308, and a monitor 302.

With reference to FIG. 7, the computer 300 performs numerical calculations at high speed in learning and inferring the CPU 316, the CPU 316, the bus 326 connected to the DVD drive 310, and the neural network in addition to the DVD drive 310. GPU 317 for storage, ROM 318 for storing boot-up programs for computer 290, ROM 318 connected to bus 326 for storing program instructions to be executed, system programs, work data during program execution, etc., and non-volatile Includes a hard disk 314 which is a non-volatile memory. The computer system 290 further exchanges data between a network I / F 304 that provides a connection to a network 328 that enables communication with other terminals and a USB memory 330 that can be attached to each part of the computer 290 and the USB memory 330. Includes a USB memory port 312 that enables The GPU 317 is for speeding up the calculation, and is not functionally essential and can be replaced by the CPU 316. However, it is desirable to have GPU317 in order to speed up the calculation.

In the present embodiment, the training data storage unit 172, the autoencoder 176, the parameter storage unit 180, the calibration parameter storage unit 186, and the like shown in FIG. 5 are all realized by the hard disk 314 or the RAM 320.

The computer program for realizing the functions of the calibration system 150 and its components in the computer system 290 is stored in the DVD 322 or the USB memory 330 mounted on the DVD drive 310, and is stored in the DVD drive 310 or the USB memory port 312 to the hard disk 314. Transferred to. Alternatively, the program may be transmitted to the computer 300 via the network 328 and stored in the hard disk 314. The program is loaded into RAM 320 at run time. The program may be loaded directly into the RAM 320 from the DVD 322 or via the network.

This program causes the computer 300 to operate as a training data collection device 170, an online calibration device training system 174, an auto encoder 178, an online calibration device 182, and a calibration device background update system 184 of the calibration system 150 of this embodiment. Contains multiple instructions. Some of the basic functions required to perform this operation are provided by an operating system (OS) running on the computer 300 or a third-party program, or modules of various programming tool kits installed on the computer 300. Will be done. Therefore, this program does not necessarily include all the functions necessary to realize the system and method of this embodiment. This program performs the above-mentioned calibration system 150 and its components. It only needs to contain the instructions. Of course, the program may include all instructions for the computer 290 to achieve the desired function. The operation of the computer system 290 is well known and will not be repeated here.

This program may be a so-called object program that can be immediately executed by CPU316, or may be a script format that needs to be converted into a format that can be sequentially executed by an interpreter.

(6) Autoencoder FIG. 8 shows a schematic configuration of the autoencoder 178 shown in FIG. With reference to FIG. 8, this auto-encoder 178 takes a time series of position data of a fixed number of time steps of two vertices as input, and has a probability distribution 354 regarding whether or not there is an edge between the two vertices. An encoder 350 that outputs a value according to p (e | ν), where ν is a time series of position data of two vertices, and e is a value indicating whether or not there is an edge between the two vertices). Input the position data at a specific time step of the combination of vertices corresponding to the largest value according to the probability distribution 354 output by the encoder 350 in response to the input of position data from the combination of different vertices. Includes a decoder 352 consisting of a neural network whose output is trained to be equal to its input. Therefore, the number of dimensions of the vector output by the decoder 352 is the same as the number of dimensions of the input vector. The online calibrator training system 174 gives the encoder 350 a time series of position data of possible combinations of vertices, and obtains the position data at a specific time point of the combination having the largest sampling value of the probability distribution 354 which is the output of the encoder 350. The operation of adjusting the parameters of the encoder 350 and the decoder 352 by giving to the decoder 352 so that the output of the decoder 352 approaches the direction equal to the input to the decoder 352 is performed from the beginning to the end of the time series of the position data. The auto encoder 178 is trained by repeating the process to be performed until a predetermined end condition is satisfied. In this embodiment, the termination condition is satisfied when the above repetition is performed a predetermined number of times.

(7) Encoder Training The encoder 350 is inspired by the neural network described in Non-Patent Document 3 that transmits a message from the vertex to the edge of the graph and further from the edge to the vertex. FIG. 9 shows an outline of the configuration of the neural network described in Non-Patent Document 3.

With reference to FIG. 9, the message transmission neural network 400 takes the combinations 390, 392, and 394 of the two vertices as separate inputs, and whether or not there is an edge between the vertices that make up each combination. It consists of a first-stage neural network 410 consisting of fully connected layers for outputting the values e ₁ , _{2, e 1, 3} and _{e 2, 3 indicating, and two of the values of the neural network 410.} With a second stage neural network 412 that takes all combinations 420, 422 and 424 as inputs and is trained to output the values 440, 442 and 444 corresponding to the three vertices input to the neural network 410. including. The neural networks 410 and 412 both consist of a fully connected layer in this embodiment.

In this embodiment, based on the configuration shown in FIG. 9, an encoder 350 having the configuration shown in FIG. 10 is used in the front stage of the autoencoder 178. With reference to FIG. 10, the encoder 350 outputs a value 440, which is output by the message transmission neural network 400 shown in FIG. 9 and the message transmission neural network 400 in response to the input of three combinations of vertices. By inputting combinations 450, 452, and combination 454, which are combinations of the two from 442 and 444, scores 470, 472, and 474, which are scores indicating the probability that an edge exists between the vertices corresponding to these combinations, are obtained. Each includes a decoder 460 for output. Scores 470, 472, and score 474 are the probabilities that vertices 1 and 2 are connected by edges, the probabilities that vertices 1 and 3 are connected by edges, and the probabilities that vertices 2 and 3 are connected by edges, respectively. The decoder 460 is trained to output such a value. The encoder 350 including the decoder 460 is trained by the online calibration device training system 174 shown in FIG. 8 by training performed on the entire autoencoder 178.

FIG. 11 shows a control structure of a computer program for operating the computer system 290 to train the autoencoder 178 in a flowchart format. With reference to FIG. 11, this program starts execution in step 500. In step 500, the position and orientation of the sensor to be calibrated are both initialized by random numbers. In step 500, the parameters of the encoder 350 and the decoder 352 are also initialized. This initialization may be performed by a random number, or may be performed by substituting a value determined by a predetermined pre-learning into each parameter. Values trained in other systems may be assigned to each parameter.

Further, this program reads the training data stored in the training data storage unit 172 shown in FIG. 5 in advance by the training data collecting device 170 from the training data storage unit 172, and loads the training data into the RAM 320 shown in FIG. 7 in step 502. including. In collecting training data, RGB-D sensors 60 and 62 are installed in the training data generation unit 160 shown in FIG. 5, and a predetermined number of people (two in this embodiment) move in a predetermined area for a predetermined time. At intervals (time steps), the training data collection device 170 collects the positions and speeds of those persons as time-series data over a predetermined time (a predetermined number of time steps) and stores them in the training data storage unit 172. It is necessary to repeat these tasks in various situations and collect many sets of training data.

This program is further trained by step 504, which follows step 502 and trains the auto-encoder 178 by repeating the process 506 a predetermined number of times for all the sets of training data in all the training data, and step 504. This includes step 508 of storing the parameters of the auto encoder 178 in a non-volatile storage device such as the hard disk 314 shown in FIG. 7 and ending the execution of the program.

The training data set referred to here is a time-series set of position data collected by the training data generation unit 160 in one training data collection session for two persons. Here, the "session" refers to a collection of measurement data for a predetermined number of time steps.

The measurement data for each time step includes, for each sensor, position data measured by that sensor for two people. Each position data includes three-dimensional coordinate data and its difference (velocity) data. That is, the position data, which is the sensor output at a certain time step, is six-dimensional. Therefore, the measurement data obtained from two sensors for two people at a certain time step includes 2 × 2 × 6 coordinate values. This coordinate value is given in local coordinates with each sensor as the origin. Since these are obtained at each time step, as a result, the measurement data of one session consists of two time series data consisting of the position data of two people output by the first sensor and two people output by the second sensor. Includes two time-series data consisting of the position data of. The set of time-series data of these four position data is referred to as a "set" of time-series data here.

The structure of this time series data is shown below in tabular form.

As can be seen from this table, the position data output by each sensor for each person in one time step is six-dimensional (position + velocity). If N time steps are measured in one session, the time series of position data output by one sensor for one person is 6 (position + velocity) dimensional vector x N, respectively. It becomes a series of. This can also be called a 6 × N-dimensional vector. Since there are two people to be measured and two sensors, the training data for one session as a whole is four 6 × N-dimensional vectors. The whole of these four 6 × N-dimensional vectors constitutes the above-mentioned “set”, and each of the 6 × N-dimensional vectors is the above-mentioned time series data.

The process 506 includes a step 510 in which the following steps 512 are repeated from the beginning of each set of training data.

In step 510, for each of the possible pairs in the set to be processed, step 520 that executes the process 522 and the value sampled from the encoder 350 as the number of pairs as a result of step 520 are compared, and the highest value is obtained. Of the pair of training data, the training data in the order specified in step 510 is selected and input to the decoder 352, and the output of the decoder 352 is calculated in response to the input in step 524. Includes step 528, which adjusts all parameters of the autoencoder 178 and ends step 512 by an error backpropagation method using an error between the input to the decoder 352 and its output.

The process 522 ends the process 522 by sampling the values output by the encoder 350 in response to the input and the step 540 in which all the training data corresponding to the pair specified by the step 520 is input to the encoder 350. Includes step 542 and. The sampled value is temporarily held in the RAM 320 shown in FIG. 7, for example.

By performing the process shown in FIG. 11, the autoencoder 178 shown in FIG. 8 is trained, and therefore the encoder 350, which is a part thereof, is also trained.

(8) Calibration Of the autoencoders 178 trained as described above, the encoder 350 is used to calibrate the acoustic processing system 162 to be calibrated in FIG. Only two RGB-D sensors 60 and 62 are shown in FIG. However, in reality, the calibration target sound processing system 162 is often provided with three or more sensors in the calibration target sound processing system 162. Further, each sensor is not limited to a three-dimensional sensor such as the RGB-D sensor 60, and may be a microphone array. If the local coordinates of any sensor are selected as the global coordinates, then any sensor may calibrate its position in pairs with the sensor corresponding to the global coordinates. The sensor selected as the global coordinates is referred to herein as a reference sensor.

The control structure of the program that causes the computer system 290 shown in FIGS. 6 and 7 to function as the online calibration device 182 shown in FIG. 5 is shown in the form of a flowchart in FIG. With reference to FIG. 12, this program includes step 560 to perform initial processing. In this initial process, the position coordinates of each sensor and its posture are initialized by random numbers. In addition, a process of securing a storage area required for the process in the RAM 320 shown in FIG. 7 is also executed in this step 560.

The program further includes step 562 of having a predetermined number of people (two in this embodiment) move within the calibration target acoustic processing system 162 shown in FIG. 5 and collect calibration data from each sensor. It is assumed that each calibration data is the same as when the encoder 350 is trained. Assuming that the time step of each training data at the time of training is m + 1, the calibration data collected in step 562 is also the m + 1 time step. In addition, calibration data can be obtained for the number of sensors.

This program further includes step 564 of calibrating each sensor by repeating the process 566 described later by the number of sensors other than the reference sensor, and storing the calibration parameters in the RAM 320 or the like.

The process 566 is a step 580 in which the following steps 582 are repeatedly executed as many as the number of possible data pairs among the calibration data to be processed, and as a result of the process in step 580, the encoder 350 is applied to each data pair. In step 584, which compares the values output from, determines whether or not each data pair points to the same person, and associates the data according to the result (identifies the person), and the type of sensor. A step 586 for calculating the calibration parameters of the position and orientation of the target sensor (sensor other than the reference sensor) using the following equation, and a step of storing the calibration parameters calculated in step 586 in the RAM 320 or the like and ending the process 566. Includes 588 and.

In step 582, step 600 for inputting all the calibration data of the pair to be processed into the encoder 350 and the value output by the encoder 350 in response to the input given in step 600 are temporarily stored in the RAM 320 or the like. Includes step 602 and the end of step 582.

When the computer system 290 executes this process, each sensor in the acoustic processing system 162 to be calibrated is calibrated. The position of each person can be determined by converting the position and posture of each sensor into the local coordinates (global coordinates) of the reference sensor using the calibration parameters obtained in this process.

The calculation formula of the calibration parameter used in step 586 is different between a three-dimensional sensor such as an RGB-D sensor and a two-dimensional sensor such as a microphone array. This formula is as described above.

(9) Parallel training with calibration In this embodiment, even after the calibration parameters are determined as described above, the calibration parameters are updated using the data output from each sensor. FIG. 13 shows the control structure of the program that causes the computer system 290 to function as the calibration device background update system 184 for that purpose.

With reference to FIG. 13, this program repeats the initial process 620 and the process 624 described later until the end condition is satisfied, thereby calculating an updated value of the calibration parameters of each sensor. The updated calibration parameters calculated for each sensor in step 622 include step 626 for updating the calibration parameters stored in the calibration parameter storage unit 186 of FIG.

In step 620, a process of securing a storage area required for processing in the RAM 320, a process of reading the calibration parameters of each sensor stored in the calibration parameter storage unit 186 shown in FIG. 5 into the RAM 320, and the like are performed.

The process 624 includes a step 640 of receiving the update data of a predetermined number of time steps from each sensor to be updated, which is similar to the training data and the calibration data, and a step 642 of repeating the process 644 described later a predetermined number of times. ..

Process 644 includes step 660 of executing the following process 662 in order from the beginning of the time series for each set of update data.

The process 662 executes the process 682 for each possible data pair in the set of data to be processed, so that the encoder 350 when all the update data related to the data bear is input for each data pair. A pair of time-series data of the decoder input is selected according to the sampling result of step 680 and the sampling result of step 680, and the data of the pair in the order specified in step 660 is input to the decoder 352 in step 684. Then, following step 684, the auto encoder 178 is subjected to an error back propagation method using the error between the input to the decoder 352 and the output of the decoder 352 obtained in step 686 and the step 686 for calculating the output of the decoder 352. Includes step 688 and step 688 to adjust the parameters.

The process 682 includes a step 700 of inputting the data for updating all the processing pairs of the set being processed into the encoder 350, and a step 702 of sampling the output of the encoder 350 with respect to the input in the step 700.

This process can operate in the background in parallel with the detection of the position of the person by the output of each sensor. Therefore, when the computer system 290 executes the program shown in FIG. 13, the computer system 290 functions as the calibrator background update system 184 shown in FIG.

2 Operation The calibration system 150 having the above configuration operates as follows.

(1) Overall operation flow The overall operation flow of the calibration system 150 is as follows.

-Sensors such as RGB-D sensors 60 and 62 are arranged in the training data generation unit 160.
-The training data collection device 170 collects training data and stores it in the training data storage unit 172.
The online calibrator training system 174 trains the autoencoder 178 using the training data storage unit 172 and the autoencoder 176. The parameters of the autoencoder 178 after training are stored in the parameter storage unit 180.

-Sensors such as RGB-D sensors 60 and 62 are arranged in the calibration target acoustic processing system 162.
-Two people walk around in the area to be detected by the RGB-D sensors 60, 62, etc., and the online calibration device 182 collects calibration data during that time.
-The online calibration device 182 reads the parameters of the autoencoder 178 from the parameter storage unit 180, and constructs the encoder 350 and the decoder 352.
-The online calibration device 182 calculates the calibration parameters of each sensor by executing the process shown in FIG. 12 on the calibration data. The calibration parameters are stored in the calibration parameter storage unit 186.

After that, using the calibration parameters stored in the calibration parameter storage unit 186, a sound source localization device or the like (not shown) executes a process of detecting the position of a person in a predetermined area.
-In parallel with the detection of the position of the person, the computer system 290 trains the autoencoder 178 in the background using the time series data obtained at that time. As a result, the autoencoder 178 including the encoder 350 is updated based on the new data.

(2) Training of the encoder 350 The training of the encoder 350 is executed as follows. With reference to FIG. 5, two people walk around in the area where the RGB-D sensors 60, 62 and the like are arranged, and the training data collecting device 170 collects the sensor output between them. This is stored in the training data storage unit 172 as training data of the autoencoder 178. When the required amount of training data can be collected, the autoencoder 178 is trained.

With reference to FIG. 11, in step 500, the positions and orientations of the sensors to be calibrated are all initialized by random numbers. In step 500, the parameters of the encoder 350 and the decoder 352 are also initialized. This initialization may be performed by a random number, or a value determined by a predetermined pre-learning may be assigned to each parameter. Values trained in other systems may be assigned to each parameter.

Further, in step 502, the computer system 290 reads the training data from the training data storage unit 172 and loads it into the RAM 320 shown in FIG. As mentioned above, this training data includes a set of training data obtained from multiple sessions. Each set contains a time series of four training data. In terms of time step units, the training data for each time step includes four 6-dimensional vectors. This is because the two sensors output 6-dimensional (position + velocity) vectors for each of the two persons.

Further, following step 502, in step 504, the autoencoder 178 is trained by repeating the process 506 a predetermined number of times for all the sets of training data in all the training data.

In step 508, the parameters of the autoencoder 178 trained in step 504 are stored in a non-volatile storage device such as the hard disk 314 shown in FIG. 7, and the execution of the program ends.

In the process 506, first, the measurement data of the first time step is selected for each set included in the training data (step 510), and the process 522 is performed for each of the data pairs possible in the set. The possible data pair in the set is a possible pair between the measurement data of the first and second persons of the first sensor and the measurement data of the first and second persons of the second sensor. To say. Taking FIG. 3 as an example, the measurement data in the first time step are x ₁₁ , y ₁₁ , x ₂₁ and y ₂₁ , and the measurement data in the second time step are x ₁₂ , y ₁₂ , x ₂₂ and y _22. Is. Of these, x ₁₁ and ₁₂ form the first time series data. This time-series data and x _1. Similarly, x ₂₁ and _{22 form} time series data x ₂ , y ₁₁ and _{12 form} time series data y ₁ , and y ₂₁ and ₂₂ form time series data y ₂ , respectively. A possible combination between these is _one of the time series data x ₁ and y ₁ observed by the sensor S 1 and one of the time series data x ₂ and y _{2 observed by the sensor S 2.} It will be a combination with one. That is, there are four possible combinations: (x ₁ , ₂ ), (x ₁ , y ₂ ), (y ₁ , x ₂ ), and (y ₁ , _2). These are represented as edges connecting the measurement data in FIG.

In step 520, the process 522 is executed for all of these four combinations. In step 540 of the process 522, for example, for _{the combination of (x 1} , ₂ ), all the training data x ₁₁ and ₁₂ constituting the time series data x ₁ and all the training data x ₂₁ constituting the time series data x ₂ A vector in which and ₂₂ are connected is input to the encoder 350. In response to this input, the encoder 350 performs an operation determined by its internal parameters and outputs one value as a result. This value is sampled from the probability distribution 354 defined by the encoder 350 in step 542, _{and indicates whether or not the time series data x 1} and the time series data x ₂ measure the position of the same person. The score. Of course, this score is unreliable, as the encoder 350 is not ready to make correct predictions at the start of training. However, by repeatedly executing the process 506, the parameters of the encoder 350 are trained, and it becomes possible to output a score indicating whether or not the input time series pair is the measurement data for the same person with high accuracy.

This score is temporarily stored in the RAM 320 as a value corresponding to the combination of _{(x 1} , _2). Similarly, the _{processing 522 is executed for the other three methods (x 1} , y ₂ ), (y ₁ , x ₂ ), and (y ₁ , ₂ ), and the value obtained in step 542 is stored in the RAM 320. do.

When the process of step 520 is completed, in step 524, the one with the highest score obtained for the above four combinations is selected, and among the time series data, the measurement data in the order specified by step 510 is combined. Is input to the decoder 352. For example, when the score of _{(x 1} , y _{2 ) is the highest, the vector in which x 11} and y ₂₁ are concatenated is input to the decoder 352.

In step 526, the output of the decoder 352 for the input vector is calculated according to the parameters of the decoder 352.

In step 528, the entire autoencoder 178 is subjected to an error backpropagation method using an error between the vector input to the decoder 352 (in the current example, _{the vector obtained by concatenating x 11} and y _{21) and the vector output by the decoder 352.} Learning is done.

Subsequently, the process of the next step 512 is performed on the measurement data of the next time step. In this case, the process performed in step 520 is exactly the same as the process performed on the measurement data of the first time step. However, the parameters of the encoder 350 are different from those of the first repetition.

The same processing is performed thereafter, but what is selected in step 524 and input to the decoder 352 is a combination of vectors of the second measurement data of the time series data having the largest value sampled in step 542. The process of step 528 is similar to that performed for the first measurement data.

In this way, when the process of step 512 is executed for all the time steps of one set of measurement data, the first process of process 506 is completed. As a result, the parameters of the autoencoder 178 change further. According to step 504, this process is repeated several more times. By repeating this process, the training of the parameters of the autoencoder 178 proceeds. Here, the end condition of step 504 is satisfied when the process of process 506 is executed a predetermined number of times, the training is completed, and the process proceeds to step 508.

In step 508, the parameters of the autoencoder 178 obtained by the iterative processing in step 504 are stored in the RAM 320, and further transferred to the parameter storage unit 180 including the hard disk 314 (FIG. 7), thereby causing the autoencoder 178 (and the encoder 350). ) Training is completed.

(3) Calibration and parallel training Calibration of each sensor in the calibration target sound processing system 162 (FIG. 5) using the encoder 350 whose training has been completed by the above processing is performed as follows.

With reference to FIG. 12, the initial process is performed in step 560. In this embodiment, in this initial process, random numbers are set for the values related to the position and orientation of each sensor. Further, the encoder 350 is constructed by reading the parameters of the encoder 350 from the parameter storage unit 180. The encoder 350 provides the same algorithm trained with the autoencoder 178.

In step 562, as many sets of calibration data as the number of sensors are collected. Here, it is assumed that the number of time steps of the calibration data is m + 1. The calibration data for this calibration is collected when two people walk around in the target area of the calibration target acoustic processing system 162 and obtain the output of each sensor at that time.

Subsequently, in step 564, the process 566 is repeated for each of the sensors other than the reference sensor. The reference sensor is a sensor whose local coordinates are determined to be treated as global coordinates, as described in the configuration. The purpose of the calibration process is to obtain parameters for converting the coordinate values of the local coordinates of other sensors into the coordinate values of the global coordinates.

In the process 566, step 582 is executed for the time series data included in the output of the sensor to be processed and the reference sensor as many as the number of possible pairs. This process is the same as the process executed in the process 522 of FIG. 11, except that the data to be processed is not the training data but the calibration data.

As a result of the process of step 582, the output of the encoder 350 is obtained for all possible pairs. The time series of the pair with the highest value among them are associated with each other as indicating the position data related to the same person. In this example, the remaining pair of time series are automatically associated with each other as indicating position data for another person.

In step 586, the calibration parameters of the position and the posture are calculated by using any of the above equations according to the type of the sensor to be calibrated by using the time series associated with each other in this way. The calibration parameters calculated in this way are stored in the calibration parameter storage unit 186 in step 588.

By executing the process of process 566 for all target sensors, the calibration parameters for determining the coordinate values obtained as the local coordinates of all sensors other than the reference sensor with respect to the coordinate values of the global coordinates are calibrated. It is stored in the parameter storage unit 186.

Further, in this embodiment, even after the calibration parameters are determined as described above, the parameters of the encoder 350 are updated using the data output by each sensor. FIG. 13 shows the control structure of the program that causes the computer system 290 to function as the calibration device background update system 184 for that purpose.

With reference to FIG. 13, when the execution of this program is started, the initial processing is performed in step 620. The initial process here includes a process of reading the parameters of the autoencoder 178 into the RAM 320, a process of securing a storage area for work in the RAM 320, and the like.

Subsequently, in step 622, the process 624 is repeated until the end condition is satisfied. In order to end this process, any condition can be used as a trigger, such as when an instruction is given from the operator or when the sound source localization device (not shown) ends the operation.

In the process 624, first, the update data collected from the sensor by a sound source localization device or the like (not shown) is received. One set of the update data shall consist of the same number of time steps as the training data.

Subsequently, in step 642, the process 644 is executed for all the update data received in step 640. The process of process 644 is almost the same as the process of training.

In the process 644, the process 662 is executed in order from the data of the first time step for each set included in the update data. At the beginning of the process 662, the process 682 is executed for each of the possible pairs for the time series data in the set to be processed. In step 700 of process 682, all update data of the pair of the pair is input to the encoder 350. In the subsequent step 702, the output of the encoder 350 is sampled and held in the RAM 320.

By executing the process 682 for all pairs in step 680, the output of the encoder 350 is obtained by sampling for each pair to be processed. In step 684, the pair with the largest value among the values sampled in this way is selected, and among the measurement data of each time step of the pair, a vector consisting of the measurement data in the order specified by step 660 is concatenated. Is input to the decoder 352. In the following step 686, the output of the decoder 352 is calculated. In step 688, the parameters of the autoencoder 178 stored in the parameter storage unit 180 are adjusted (updated) by the error back propagation method using the error between the input to the decoder 352 and the output from the decoder 352.

Then, when the predetermined end condition is satisfied, the program ends the execution. While this program is running, the parameters of the autoencoder 178 are updated in the background. Since the encoder 350 is not used for the operation of the sound source localization device or the like (not shown), updating the parameters of the autoencoder 178 in this way does not affect the operation of the sound source localization device or the like. Next time, the operation of the encoder 350 when the online calibration device 182 performs the calibration process will change.

As described above, according to the calibration system 150 according to this embodiment, after the training data is automatically generated, the autoencoder 178 (and the encoder 350) is automatically trained without any human intervention. Also, when calibrating the sensors included in the calibration target sound processing system 162, two people simply walk within a predetermined area to generate calibration data without manually setting the actual position and posture of each sensor. Therefore, the calibration parameters of each sensor can be calculated automatically. Further, there is an effect that the calibration parameter can be automatically calculated not only by the 3D sensor such as the RGB-D sensor 60 but also by the sound processing system in which the 2D sensor and the 3D sensor such as the microphone array are combined.

3 Experiment In order to test the performance of the calibration process by the autoencoder 178 according to the first embodiment, the experiment described below was performed. In the experiment, the open data set used in Non-Patent Document 4 was used. Gaussian noise with a standard deviation of 15 cm was added to each camera measurement. The initial position of the microphone array was randomly set and the target angle with respect to the microphone array was generated by Gaussian noise with a mean of 0 and a standard deviation of 2.

First, a set of five measurement data was prepared for training the autoencoder 178. Each set contained a set of time series data with 100 time steps.

As a result of performing a person identification process using the encoder 350 trained in this experiment, the rate at which the computer 300 correctly determined that different persons were different was 98.3%. As a result of calibration using this computer 300, the average error obtained was 22 mm for the RGB-sensor and 57 mm for the microphone array.

Thus, according to this first embodiment, both the RGB-sensor and the microphone array can be calibrated with high accuracy without manual work. Similarly, the encoder 350 required for calibration can be trained without human intervention. This training is unsupervised learning and does not require manual training data preparation. In both the preparation of training data and the calibration process, it is only required that a predetermined number of people walk around a predetermined area and speak appropriately when a microphone array is included in the target.

2. Second Embodiment 1 Configuration (1) Overall Configuration The second embodiment relates to an online calibration device training system. Unlike the online calibrator training system 174 shown in FIG. 5, the online calibrator training system according to the second embodiment trains the autoencoder 178 by the backpropagation method by a mini-batch.

(2) Encoder Training FIG. 14 shows a control structure of a program for training the auto encoder 178 in the second embodiment (a program for making the computer system 290 function as an online calibration device training system) in a flowchart format. .. With reference to FIG. 14, this program differs from that shown in FIG. 11 in that, instead of step 504 in FIG. 11, process 722 includes step 720, which repeats process 722 for all training data a predetermined number of times. ..

The process 722 includes a step 740 that divides all the training data into a predetermined number of mini-batch, and a step 742 that executes the process 746 in order from the first mini-batch among these mini-batch.

Step 742 includes step 760 for accumulating the error obtained by each training data in the mini-batch by executing the process 746 in order from the beginning of each set of training data in the target mini-batch, and step 760 and the error accumulated by step 760. Includes step 764, which adjusts the parameters of the autoencoder 178 by the backpropagation method and ends the process 746.

The process 746 executes the process 762 in order from the beginning of each set of training data of the mini-batch to calculate the accumulated error for the mini-batch, and the error backpropagation using the error accumulated in the mini-batch. Includes step 764, which adjusts the parameters of the autoencoder 178 by method to end process 746.

The process 762 sets the highest value among the scores sampled in step 780 and step 780, which execute the process 782 for sampling the score output by the encoder 350 for each pair of time-series data of the set to be processed. Step 784 of inputting the measurement data (of the time step) in the order specified by step 760 of the time series data corresponding to the corresponding pair into the decoder 352, and the output of the decoder 352 in response to the input in step 784. Includes step 786 to calculate and step 788 to accumulate processing 762 by accumulating errors between the output and input of the decoder 352.

The processing 782 samples and stores the output of the encoder 350 corresponding to the input of step 800 and the step 800 of inputting the full position data of the pair of the time series data of the processing target of the processing target set to the encoder 350. Includes step 802 to end process 782 for the pair to be processed.

2 Operation (1) Overall flow of operation The overall flow of operation during training of the online calibration device training system according to the second embodiment is different from the operation of the first embodiment of the training data. The point is that the backpropagation method is applied not for each set but for each mini-batch. In other respects, the online calibrator training system according to the second embodiment and the online calibrator training system 174 according to the first embodiment operate in the same manner.

(2) Encoder Training With reference to FIG. 14, the online calibration device training system according to the second embodiment performs initial processing in step 500. Subsequently, in step 502, the training data is read from the training data storage unit 172 (see FIG. 3) and loaded into the RAM 320 (see FIG. 7).

Subsequently, the computer system 290 repeats the process 722 for all the training data a predetermined number of times.

In each repetition of the process 722, all the training data is divided into mini-batch (step 740), and the process 746 is executed in order from the first mini-batch. This process adjusts the parameters of the autoencoder 178.

In the process 746, the error related to the mini-batch is accumulated by executing the process 762 in order from the beginning of each set of training data included in the mini-batch to be processed (step 760).

Specifically, in the process 746, for each of the possible pairs of the time-series data that first constitutes the set, all the position data that constitutes the time-series data of the set is input to the encoder 350 (step 800). , A score indicating whether or not the persons represented by the pair of time series data input to the encoder 350 are the same person is sampled (step 802). By executing this process, scores are calculated for all possible pairs of time-series data that make up the set.

Subsequently, in step 784, the time-series pair corresponding to the highest score among the scores calculated in step 760 is selected, and the order specified in step 760 from each of the time-series data constituting the pair. The position data of the time step of is selected and input to the decoder 352 (step 784). In step 786, the output of the decoder 352 is calculated, and in step 788, the error between the input and the output of the decoder 352 is calculated and accumulated.

By executing the process 762 in this way, errors are accumulated for the mini-batch to be processed specified in step 742. Using this error, the parameters of the autoencoder 178 are adjusted by the error backpropagation method in step 764. At this time, the accumulation error of the mini-batch is cleared.

By executing the process up to step 764 for all the mini-batch sequentially selected in step 742 in this way, the training of the autoencoder 178 using all the training data is completed once. By repeating this training a number of times specified in step 720, the training of the autoencoder 178 is completed. The parameters of both the encoder 350 and the decoder 352 of the autoencoder 178 thus obtained are stored in the parameter storage unit 180 of FIG.

In this way, the training of the online calibration device training system according to the second embodiment is completed, but the online calibration device training system also calibrates the sensor in the same manner as the online calibration device 182 according to the first embodiment. Can be executed.

The embodiments disclosed this time are merely examples, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by each claim of the scope of claims, taking into consideration the description of the detailed description of the invention, and all changes within the meaning and scope equivalent to the wording described therein. include.

50 Sensor system 60, 62 RGB-D sensor 64 Microphone array 66, 68 Target person 80, 82 Local coordinates 100 Graph 120, 122, 124, 126 Edge 150 Calibration system 160 Training data generator 162 Calibration target sound processing system 170 Training Data collection device 172 Training data storage unit 174 Online calibration device Training system 176, 178 Auto encoder 180 Parameter storage unit 182 Online calibration device 184 Calibration device Background update system 186 Calibration parameter storage unit 290 Computer system 300 Computer 302 Monitor 304 Network I / F
306 Keyboard 308 Mouse 310 DVD Drive 312 USB Memory Port 314 Hard Disk 316 CPU
317 GPU
318 ROM
320 RAM
322 DVD
326 Bus 328 Network 330 USB Memory 350 Encoder 352, 460 Decoder 354 Probability Distribution 390, 392, 394, 420, 422, 424, 450, 452, 454 Combination 400 Message Transmission Neural Network 410, 412 Neural Network 440, 442, 444 value 470, 472, 474 score

Claims (14)

It is a calibration device for calibrating the positions and orientations of the first sensor and the second sensor, each of which can detect and output the positions of a plurality of moving objects in a discrete time series.
An acquisition unit that acquires first time-series data and second time-series data regarding the positions of a predetermined number of moving objects, respectively, measured by the first sensor and the second sensor over a predetermined time.
With the first time-series data and the second time-series data as inputs, the first moving body represented by the first time-series data and the second time-series data represented by the second time-series data. For each combination with the moving body, the time series data of the position of the first moving body in the first time series data and the time series of the position of the second moving body in the second time series data. A neural network pre-trained to take data as input and output a score indicating whether the first and second moving bodies forming the combination are the same moving body. Mobile body identification means consisting of
Based on the output of the moving body identification means, the correspondence between each moving body represented by the first time-series data and each moving body represented by the second time-series data is estimated, and the correspondence is estimated. Using the relationship, the position and orientation of the second sensor with respect to the first sensor are calibrated so that the output error between the first sensor and the second sensor for each moving body satisfies a predetermined condition. A calibrator, including a sensor calibrator. The calibration device according to claim 1, wherein the sensor calibration means includes a minimization means for calibrating the position and orientation of the second sensor so as to minimize the sum of the output errors. Further, claim 1 includes a parallel training means that uses the first time-series data and the second time-series data to train the moving body identification means in parallel with the operation of the correspondence estimation means. Alternatively, the calibration device according to claim 2. The parallel training means
With the mobile body identification means
A decoder that inputs the output of the mobile body identification means and the position data of each of the first time series data and the second time series data at the same time step.
The moving object identification means and the above means so that the output of the decoder is close to the position data of the same time step input to the decoder over a predetermined range of the first time series data and the second time series data. The calibrator according to claim 3, further comprising an adjusting means for training the moving body identification means by adjusting parameters with the decoder. The adjusting means uses the first time-series data and the second time-series data over the entire predetermined time to obtain the output of the decoder and the position data of the same time step input to the decoder. The calibration device according to claim 4, further comprising an error back-propagation means for training the moving body identification means by adjusting parameters between the moving body identification means and the decoder by an error back-propagation method using an error. The parallel training means trains the moving body identification means each time the first time series data and the second time series data are given, according to any one of claims 3 to 5. The calibrator described. The first time-series data and the second time-series data regarding the positions of the first moving body and the second moving body, respectively, measured by the first sensor and the second sensor over a predetermined time. As an input, a mobile body identification device comprising a neural network pre-trained to output a score indicating whether or not the first mobile body and the second mobile body are the same mobile body. Each of the first time-series data and the second time-series data includes position data of the target moving body at predetermined time intervals.
The moving body identification device according to claim 7, wherein each of the position data for each predetermined time includes the position and speed of the target moving body and time information indicating the time when the position and speed are measured. .. The neural network receives a plurality of inputs that receive the position and velocity included in the first time series data, and the position and velocity included in the second time series data, and an output that outputs the probability. The moving body identification device according to claim 8, which is a neural network composed of a plurality of layers having the above. A time-series data acquisition unit that acquires a time-series of position data obtained in a predetermined time step over a predetermined time for each of a plurality of moving objects.
A position data extraction means for extracting position data acquired at the same time in a specified order from the time series of the position data acquired by the time series data acquisition unit.
A first neural network with an input determined by the number of predetermined time steps and at least one output.
A second neural network, both of which have the same number of inputs and outputs as determined by the position data that make up the time series.
All possible combinations of the two moving bodies are extracted from the plurality of moving bodies, and among the time series of the position data, the time series of the position data of the moving bodies constituting the extracted combination is the time series of the first neural network. -The input means given to the first neural network as the input to the network and
A first sampling means that samples the value output by the first neural network in response to the input.
A selection means for selecting the combination in which the largest value is obtained from the values sampled by the first sampling means for each of the possible combinations, and
Among the position data extracted by the position data extraction means, the position data of two moving bodies corresponding to the combination selected by the selection means is input to the second neural network, and the second neural network is used. -A second sampling method that samples the output of the network,
The error between the two moving body position data given to the input of the second neural network and the value sampled by the second sampling means from the output of the second neural network is reduced. In addition, a parameter adjusting means for adjusting the respective parameters of the first neural network and the second neural network by the error back propagation method, and
The position data extraction means, the first neural network, the input means, the first sampling means, the selection means, the second sampling means, and the parameter adjusting means are used in a time series of the position data. A first repetitive execution means that specifies position data in order from the beginning and repeatedly operates until the time series data ends.
A second repeat execution means that repeatedly executes the repetition by the first repeat execution means until a predetermined end condition is satisfied.
A training device including a parameter storage means for storing the parameters of the first neural network at a time when the repetition by the second repetition execution means is completed in a predetermined storage device. The parameter adjusting means is between the two moving body position data given to the input of the second neural network and the value sampled by the second sampling means from the output of the second neural network. An error accumulating means that accumulates the error of the above a predetermined number of times,
After the first sampling means and the second sampling means have operated a predetermined number of times, the first neural network is used by an error backpropagation method so that the error accumulated by the error accumulating means is reduced. The training apparatus according to claim 10, further comprising a batch adjusting means for adjusting each parameter of the network and the second neural network by batch processing. A computer program that causes a computer to function as the calibration apparatus according to any one of claims 1 to 6. A computer program that causes a computer to function as the mobile identification device according to any one of claims 7 to 9. A computer program that causes a computer to function as the training device according to claim 10 or 11.

Images