JP7556098B1 - Movement line estimation method and movement line estimation system - Google Patents

Abstract

[Problem] To provide a method for estimating a user's position more accurately based on inertial information measured by a user terminal even in a real environment. [Solution] A computer system executes a step of generating a first movement trajectory (30) based on inertial information output by an inertial measurement device (10), a step of acquiring floor information (40b) describing the layout of a movement space (40), a step of acquiring visited positions (42, 43) visited by the user in the movement space, a step of setting a floor plan (40a) including the floor information and the visited positions, and a step of outputting a second movement trajectory (60) by correcting the first movement trajectory based on the floor plan, in which the user position acquisition step acquires a visited position with an unknown time and a visited position with a time attached thereto, and the trajectory correction step corrects the first movement trajectory based on a movable area in the movement space, the position of an obstacle (41) that cannot be moved, and the visited position. [Selected Figure] Figure 1

Description

The present invention relates to a method for estimating traffic flow and an information processing system. More specifically, the present invention relates to a method for estimating traffic flow using an inertial measurement unit and a computer system that performs information processing related to traffic flow estimation.

In a real environment (also called the "real world"), measuring or estimating people's movements (sometimes called "movement trajectories") can be used to evaluate the effectiveness of advertising, for example by determining whether customers who actually visit a retail store purchase a product advertised in the store or linger in front of the shelf on which the product is placed. Alternatively, estimating the movements of customers and store staff can be used to develop and improve the layout (floor plan) of the store. In patrol inspections of facilities such as factories, understanding the time that inspectors spend at each inspection location, the time taken from the start to the end of the inspection and the relationship between the route, etc. can be used to improve the inspection route.

Known technologies for measuring and estimating people's movement paths (hereinafter referred to as "movement path estimation technologies") include those that use AI cameras and beacons. As is well known, AI cameras estimate movement paths by detecting the user whose movement path is to be estimated and the direction of movement of that user based on footage captured by a surveillance camera or the like. However, with movement path estimation technologies using AI cameras, it is difficult to estimate movement paths over a wide area that exceeds the camera's shooting range. In addition, all people who enter the camera's field of view are recorded, regardless of whether they have consented to being photographed by the camera. In particular, there are issues with people who do not consent, such as violation of privacy and portrait rights, or being unable to enter the environment where the movement path estimation is being performed or to enjoy the services provided in that environment.

A beacon is a wireless signal that uses Bluetooth Low Energy (BLE) technology to communicate with a user terminal such as a smartphone carried by the user. When the user terminal detects a beacon, it reads the identifier of the transmitting device contained in the beacon and executes a predetermined process. In this movement line estimation technology using a beacon, application software that executes a predetermined process to estimate a movement trajectory when a beacon is received is installed on the user terminal. When the user terminal receives a beacon, it acquires information such as the beacon's signal strength and the distance to the transmitting device, and identifies the reception position. Then, the user's smartphone receives the beacon at each location where a transmitting device is installed, and the movement line of the user is estimated. However, there is a limit to the range that a beacon can reach, and it is difficult to estimate movement outside the beacon's receivable range. In addition, in order to estimate the movement line with high accuracy, it is necessary to install more beacon transmitting devices. Note that the following Patent Document 1 describes one estimation technology using a beacon.

On the other hand, the technology of estimating the line of movement using an inertial measurement unit (hereinafter sometimes referred to as "IMU"), which is the subject of the present invention, is called inertial navigation or inertial localization (hereinafter sometimes referred to collectively as "inertial localization technology"). The IMU is composed of an accelerometer and a gyroscope, and measures acceleration and angular velocity in three axial directions (three dimensions). Inertial localization technology has been researched mainly in the field of robotics, for example, but it can also be applied to estimating the line of movement of a user by using an IMU installed in a user terminal (smartphone, smart watch, tablet terminal, etc.). Therefore, inertial localization technology does not require the installation of devices necessary for line of movement estimation, such as surveillance cameras and beacon transmitters, in the real environment. Inertial position estimation technology continues to measure acceleration and angular velocity as long as the user carries the user terminal, making it possible to estimate the user's movement trajectory over a wide area. In recent years, a method that uses the IMU of the user terminal to learn the relationship between the inertial information measured by the user terminal and the actual movement trajectory using a neural network has been attracting attention in order to estimate the position and movement line with higher accuracy (see, for example, Non-Patent Documents 1 and 2).

The following non-patent document 3 describes "Scaled Dot-Product Attention" related to an embodiment of the present invention. In addition, non-patent document 4 describes a method of using a service that allows users to scan product barcodes with their smartphones in stores and pay at a dedicated register in relation to an application example of the present invention. Non-patent documents 5 and 6 describe application software that is installed on smartphones to use the service. Non-patent document 7 describes an example of a shopping cart called a "smart cart" that is equipped with a barcode reader and does not require payment at a register. Non-patent document 8 describes the current trends of smart carts. In addition, the following patent document 2 describes a method of determining the current position of a pedestrian by performing map matching based on the context using the detection values of sensor units such as IMUs and map information.

Patent No. 6695064 JP 2020-085783 A

Yan, H., Herath, S. and Furukawa, Y.: RoNIN: Robust neural inertial navigation in the wild: Benchmark, evaluations, & new methods, ICRA (2020) Herath, S., Caruso, D., Liu, C., Chen, Y. and Furukawa, Y.: Neural inertial localization, CVPR (2022) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I.: Attention is all you need, NeurIPS (2017). AEON Co., Ltd., "How does "RegiGo" work, where you don't have to line up at the cash register? A detailed explanation of how to use it and how to pay!", [online], [Searched June 23, 2023], Internet <URL: https://www.aeon.co.jp/column/20230203/> AEON Co., Ltd., "Regigo", [online], [searched June 23, 2023], Internet <URL: https://www.regigo.jp/> Bussan Food Service Co., Ltd., "Business content", [online], [searched June 23, 2023], Internet <URL: https://www.bussanfood.jp/business/> Trial Holdings Co., Ltd., "Trial Regi Cart", [online], [searched June 23, 2023], Internet <URL: https://www.trial-net.co.jp/prepaid/regi-cart/> Frontier Management Co., Ltd., "Transformation brought about by "caper" smart carts (Retail Tech) Notable technology of overseas startup companies 1", [online], [Retrieved June 23, 2023], Internet <URL: https://frontier-eyes.online/caper_retail-tech-series_featured_technology_of_abroad_start-up-companies01/>

A user terminal equipped with an IMU inevitably measures unnecessary inertial information (noise) associated with the user's walking and various operations on the terminal. Inertial position estimation technology is based on the basic principle of estimating a movement trajectory from a starting point to a current position based on a distance obtained by integrating twice the acceleration measured during movement, starting from the measurement start position of the inertial information, and a posture obtained by integrating a displacement angle obtained by integrating the angular velocity. Therefore, even if the noise is minute, there is a problem that an error between the true movement trajectory and the estimated movement trajectory is accumulated. This problem is also present in the method described in Patent Document 2. In addition, the method described in Patent Document 2 uses map information to determine the position of a pedestrian assuming that the pedestrian is present on a walkable passage, so it is difficult to accurately determine the position of a pedestrian in an open area where there is no passage or the passage is wide.

In the inertial position estimation technology described in References 1 and 2, a neural network model that estimates movement lines from inertial information containing noise is obtained by supervised learning, and then the inertial information measured by the IMU is applied to the model. This reduces errors caused by noise. However, the conditions of the movement space (layout, area, etc.) differ between the environment when the neural network is learning and the real environment. In addition, it goes without saying that the unspecified number of users who are the targets of actual movement estimation are not test subjects who are aware of the intention of the test at the time of learning. Therefore, there is actually a large discrepancy between the actual movement lines of users in the real environment and the movement lines estimated by the neural network based on the inertial information measured by the user's terminal.

The present invention aims to provide a movement line estimation method that can more accurately estimate a user's position and movement line based on inertial information measured by the user's terminal even in a real environment, and a movement line estimation system that is a computer system for estimating movement lines based on the method.

One aspect of the present invention for achieving the above object is a method for estimating a user's movement line in a moving space using a computer system, comprising:
The computer system comprises:
a trajectory generating step of generating a first movement trajectory based on inertial information outputted by a user terminal equipped with an inertial measurement device and moving in the movement space together with the user;
a floor information acquisition step of acquiring floor information describing a layout of the movement space;
a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space;
a floor plan setting step of setting a floor plan including the floor information and the visiting location;
a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory;
Run
In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired,
In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position.
This is a method for estimating traffic flow.

The moving space is a store,
The computer system is linked to a POS system of the store;
A product information acquisition step of acquiring information on a product purchased by a user via the POS system,
The floor information includes the locations of product shelves in the store, information on products displayed on each product shelf, and the location of a place where a predetermined operation for a POS system is performed when making a payment,
The visiting location acquisition step can also be a traffic flow estimation method in which the visiting location with an unknown time based on the product information acquired in the product information acquisition step and the visiting location with a time attached based on a specified operation time for the checkout are acquired.

Furthermore, the floor information may include a recording start position at which the recording of inertial information is started by the user terminal, and the method may be a traffic flow estimation method in which the visited position with a time attached is obtained based on the time at which an instruction to start recording the inertial information is received by the user terminal and the recording start position.

The moving space is a facility where a patrol inspection is performed,
The floor information includes an inspection location in the facility,
The user position acquisition step may be a traffic flow estimation method in which the inspection position is acquired as the visiting position, and the visiting position includes the visiting position accompanied by a time based on user input to a specified device at a specified inspection position.

In the trajectory generation step, a trained neural network may estimate the first movement trajectory.

In any of the above traffic flow estimation methods, the visited locations with associated time may include images captured by an AI camera and/or images obtained based on a beacon received by the user terminal.

The scope of the present invention also includes a computer system that is configured with one or more information processing devices and estimates a user's movement line in a moving space, and the computer system includes:
The information processing device includes a user terminal equipped with an inertial measurement device and moving in the moving space together with the user;
storing floor information describing the layout of the travel space;
a trajectory generating step of generating a first movement trajectory based on inertial information output by the user terminal;
a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space;
a floor plan setting step of setting a floor plan including the floor information and the visiting location;
a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory;
Run
In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired,
In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position.
It is a traffic flow estimation system.

The present invention provides a movement line estimation method and a movement line estimation system that can estimate a user's position more accurately based on inertial information measured by the user's terminal even in a real environment. Other advantages will be explained below.

FIG. 2 is a diagram illustrating a concept of a flow line estimation method according to an embodiment of the present invention. 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. 11A and 11B are diagrams illustrating results of a comparative experiment between a flow line estimation method according to an embodiment and a flow line estimation method according to a comparative example. 1 is a diagram showing a procedure for estimating a movement trajectory of a user in a store using a flow line estimation method according to an embodiment. 11 is a diagram specifically illustrating information processing, communication flow, and the like when the traffic line estimation method according to the embodiment is applied to a store. FIG.

An embodiment of the present invention will be described with reference to the accompanying drawings.
==== Working Example ====
The technical idea of the present invention is to use a computer system to correct an inaccurate movement trajectory obtained by an existing inertial position estimation technology (for example, Inertial Navigation described in Non-Patent Document 1 above) based on information describing the situation (size, shape, arrangement of obstacles, etc.) of the user's movement space, which is an individual test environment including a real environment, and the positions of places actually visited by the user in the information, without relying on machine learning. Note that, hereinafter, information describing the situation of the movement space will be referred to as "floor information", and information in which the floor information is accompanied by the positions of places actually visited by the user (hereinafter sometimes referred to as "visited positions") will be referred to as "floor plan" or "floor plan information".
<Concept of the flow line estimation method according to the embodiment>
FIG. 1 is a diagram showing the concept of a flow line estimation method according to an embodiment based on the above technical idea. As shown in FIG. 1, the basic principle of the inertial position estimation technology is to estimate a user's movement trajectory (flow line) based on inertial information measured by an IMU 10 mounted on a user terminal 1 such as a smartphone carried by a user. In the flow line estimation method according to the embodiment, first, a movement trajectory 30 is obtained using an existing inertial position estimation technology, for example, a trained neural network 20 described in the above-mentioned non-patent documents 1 and 2. However, the movement trajectory 30 estimated using the existing inertial position estimation technology in a test environment different from that during learning has many errors compared to the actual trajectory.

The movement trajectory 30 obtained using the neural network 20 is corrected based on a floor plan 40a of the movement space 40 of the user that is the subject of the traffic line estimation, such as an arbitrary test environment or real environment. The floor information 40b constituting the floor plan 40a is data that is a so-called "blueprint" of the movement space 40, and is information including the planar shape and size of the movement space 40, and obstacles 41 (pillars, walls, fixed furniture and fixtures, etc.) installed within the movement space 40. The area where the obstacles 41 are located is a position where the user cannot be present, and the actual movement trajectory is within the movement space 40 and avoids the obstacles 41.

In summary, the method for estimating the movement path according to the embodiment estimates an accurate movement path 60 of the user in the movement space 40 by applying the inertial information continuously measured by the IMU 10 to the neural network 20 described in the cited document 1 to estimate an (inaccurate) movement path 30, and applying the inertial information to a neural network 50 that corrects the (inaccurate) movement path 30 using discrete information as clues, such as a visited location 42 shown by a white circle in the figure, which is a location where the user actually stopped (can stop, will stop) at an unknown time, a visited location 43 shown by a black circle in the figure, whose time is known, and the location of an obstacle 41 where the user cannot stop,

The traffic line estimation method according to the embodiment can be applied to, for example, a case where the sales floor of a store is treated as the movement space 40, and the movement trajectory 60 of a user (customer) moving within the store is estimated. The floor information 40b corresponding to the sales floor of the store includes, in addition to structures such as pillars and walls, product shelves, cash registers, etc. as obstacles 41. This floor information 40b is updated from time to time in response to product replacement, renovation, etc., and the latest floor information 40b is used when estimating the movement trajectory 60.

Meanwhile, the POS (Point of Sales) system in the store can obtain the purchased items of the user who purchased the product at the store. Then, by checking the purchased items against the floor information 40b, the location of the shelf where the product purchased by the user is placed can be specified as the location (visited location) 42 where the user actually visited, and information on the visited location with unknown time can be obtained. Furthermore, since the store involves a process of payment, the location and time of which can be specified, a visited location 43 with a known time can be specified based on the location (POS terminal, cash register, etc.) where a specific operation is performed on the POS system during payment and the time of payment, and a visited location with a time attached can be generated. This makes it possible to set a floor plan 40a consisting of the floor information 40b and the visited locations (42, 43) of the user.
<Calculation method for estimating traffic flow>
The inertial position estimation technology estimates the movement trajectory of a user based on inertial information measured by the IMU 10. The calculation method for the estimation is as follows: first, a normalized finite two-dimensional space Ω=[0,1] ² is considered as the movement space 40, which is the environment in which the user is active, and the movement trajectory P of the user in a certain time interval [1,T] is expressed as a series of two-dimensional positions, P=( _P1 ,..., _PT ), _Pt∈Ω . On the other hand, six-dimensional inertial information M consisting of three-dimensional accelerations and three-dimensional angular velocities obtained in time synchronization from a device (smartphone, ^smartwatch , etc.) equipped with the IMU 10 that the user freely operates is expressed as M=( _m1 ,..., _MT ), _mt∈R6 .

In a conventional flow line estimation method based on an existing inertial position estimation technology, a neural network that estimates a movement trajectory P from inertial information M is obtained by supervised learning from previously collected data D = {M _i , P _i } ^N _{i = 1.} For example, Non-Patent Document 2 describes Neural Inertial Localization that obtains a neural network (hereinafter, sometimes referred to as a "correction network") that corrects an error Δ between a movement trajectory Q obtained by Neural Inertial Navigation described in Non-Patent Document 1 and an actual movement trajectory P. In this Neural Inertial Localization, the moving speed of the user in the world coordinate system is estimated from the inertial information M using the trained neural network shown in Non-Patent Document 1, and the error Δ = (δ ₁ , ..., δ _T ), δ _t = P _t - q _t is predicted (corrected) so that the movement trajectory Q = (q ₁ , ..., q _T ) obtained by integrating the moving speed matches the actual trajectory P.

The Neural Inertial Localization described in Non-Patent Document 2 has an advantage that, once a correction network for predicting the error Δ is obtained, the movement trajectory P can be estimated only from the inertial information M. However, as described above, in conventional movement line estimation methods including Non-Patent Document 2, when the movement space 40, the situation in the movement space, the user, etc. are different between the learning environment in which the data D is collected and the test environment or the actual environment, there is a problem that the estimation accuracy of the estimated movement trajectory P decreases. Therefore, in the movement line estimation method according to the embodiment, the learning data D is not collected in advance, but the floor plan information A=(X, _Xtimed , _Ωobs ) obtained from an individual test environment is used to optimize the correction network for the movement trajectory Q based on the inertial information M.

Note that X in floor plan information A is a visited location 42 with an unknown time, and is given as a set of absolute locations _xk of places that are known to have been definitely visited by a user, although the specific time is unknown. For example, in the above-mentioned application example to a store, the location of a product shelf where a purchased item is placed, which is known by comparing the purchase history by the POS system with the floor information 40b, and the location of an inspection target in the above-mentioned patrol inspection work, etc., can be mentioned. The visited location X with an unknown time is expressed as X = ( _xj ) ^Jj _{= 1} , _xj ∈ Ω.

_Xtimed in the floor plan information A is a visited location 43 with a time attached, and is composed of an absolute location _xk of a place that the user has definitely visited and its visiting time _tk . It is a self-checkout, a checkout counter when a payment is settled by electronic payment, an automatic ticket gate, etc., and is a pair of the location of the user terminal 1 equipped with the IMU 10 operated by the user and the usage time. The visited location _Xtimed with a time attached is expressed as _Xtimed = {( _xk , _tk )} ^Kk ₌₁ , ( _xk , _tk ) ∈ Ω×N.

Ω _obs in the floor plan information A is a set of positions to which the user is not likely to move regardless of time, that is, the above-mentioned obstacle area (hereinafter, sometimes referred to as the “obstacle layout”). The obstacle layout Ω _obs is expressed as Ω _obs ⊂ Ω.

Here, it is considered that the visited position X _timed can only be acquired at a few points for a movement of several hundred meters. In addition, the inertial information M from the IMU 10 can generally be acquired at any time at a period of several tens to several hundreds of Hz. Then, in addition to the estimated result of the movement trajectory Q obtained by the trained model described in Non-Patent Document 1, the visited positions X and X _timed in the test environment are input to estimate the correction value Δ=(δ ₁ , . . . , δ _T ). The estimated result Q is calculated by integrating the speed component estimated from the inertial information M, and since errors in the speed estimation accumulate, the correction value δ _t is considered to depend on the time t. In addition, some of the estimated positions q _t should be corrected in the direction approaching the visited position XεX _timed , and it is desirable that the relationship between X and X _timed can be described in the correction network. Then, under the above assumptions, a correction network expressed by each of the formulas (1) to (3) in the following "Mathematical Expression 1" was designed.

Here, _Wq , _Wk , _Wv ∈ ^R2×d , H∈ ^R×d , and X∈ R ^{(|X|+|Xtimed|)×2} _is a matrix in which the elements of X and _Xtimed are stacked in the row direction. attn( ^QTWq , _XWk , _XWv ) is the Scaled Dot-ProductAttention described in Non-Patent Document 3, and cat( _qt , _ht , t) is a combination of multiple vectors. And, MLP is a multilayer perceptron, and is composed of a linear layer and a ReLU (Rectified Linear Unit) function.

In the correction network designed as above, the movement trajectory Q is corrected to P'=Q+Δ, _p't = _qt + _δt . For this correction result, an objective function L(P',A) using floor plan information A is defined, and the parameters of the correction network are minimized by using the gradient descent method. Specifically, L is given as in equation (4) in "Equation 2" below. Furthermore, _Lvp , _Lobs , _Lbd , and _Lreg included in equation (4) are respectively expressed by equations (5) to (8) in "Equation 2" below.

In equation (4) of the above "Mathematical Expression 2," L _vp shown in equation (5) is a term related to the positional relationship between P' _t and visited position _{X∈X∪Xtimed} . By minimizing this L _vp , when the corresponding time for each visited position x is known, the estimated value p' _t of that time can be brought closer to x, and when the time is unknown, p' _t at the position closest to the current position x can be brought closer to x.

In equation (4), L _obs shown in equation (6) is a term related to the positional relationship with an obstacle, and the loss becomes large if the corrected position p' _t is above an obstacle. And d _obs in L _obs is a function that gives the shortest distance to an obstacle when p' _t is inside any obstacle, and gives 0 when p' t does not collide with any obstacle.

On the other hand, in formula (4), _dΩ defined in L _bd shown in formula (7) calculates the distance (squared) to the nearest boundary as a loss when p' _t exists outside the environment Ω = [0, 1] ^2. As a result, L _bd has a large loss when the corrected position p' _t is outside the environment. In formula (4), L _reg shown in formula (8) is a regularization term, which suppresses the position p' _t from changing suddenly due to the correction δ _t . λ _vp , λ _obs , λ _bd , λ _reg are weights for each term, and can be set appropriately based on experience, etc.

In addition, W _t in L _vp in equation (5) is a weight determined from the time of x and p' _t , and is defined as in equation (9) in "Mathematical formula 3" below.

In equation (9) shown in "Mathematical Expression 3", the first line indicates that for a visited position x, if the time t' is known, then the position _p't at the same time is moved closer to x, the second line indicates that if the time is not known, then nothing is done, and the third line indicates that for a visited position x, if the time is unknown, then the position p' close to the visited position x is moved closer to x.
<Experiment and Evaluation>
Next, in order to confirm the effect of correcting the movement trajectory by the above-mentioned correction network, the Inertial Localization Dataset (hereinafter sometimes referred to as the "dataset") described in the above-mentioned non-patent document 2 was used as experimental data. The dataset is composed of three different environments: University A (university campus A), University B (university campus B), and Office. University A and University B are outdoor environments, and Office is an indoor environment. The size of each environment is 62.8 × 84.4 ^m2 for University A, 57.6 × 147.2 ^m2 for University B, and 38.4 × 11.2 ^m2 for Office. The dataset was acquired by a user moving around each environment while operating two smartphones equipped with IMUs 10 as user terminals 1.

In general, one of the two smartphones (hereafter referred to as "smartphone A") is held in the hand and records the actual movement trajectory P using Visual Inertial SLAM. The other smartphone (hereafter referred to as "smartphone B") is freely used by the participant, for example by holding it in the hand, putting it in a pocket, or making a phone call. Note that in both smartphones, inertial information M is recorded in time synchronization with the movement trajectory P. The above dataset includes the movement trajectories and all sequences of inertial information M recorded by the two smartphones.

Then, using the above dataset, a movement trajectory was estimated by the movement line estimation method according to the embodiment, and an experiment was conducted to compare the movement trajectories estimated by the embodiment and the comparative example, using a movement line estimation method using a conventional inertial position estimation technology as a comparative example. As the comparative examples, a method similar to Neural Inertial Navigation described in Non-Patent Document 1 (hereinafter, sometimes referred to as "Comparative Example 1") and a method similar to Neural Inertial Localization described in Patent Document 2 (hereinafter, sometimes referred to as "Comparative Example 2") were used. In addition, when estimating the movement trajectory P by the embodiment and comparative examples 1 and 2, 1/6 of all sequences in the above dataset was used as test data. The remaining 5/6 was used as learning data for the neural network in comparative examples 1 and 2.

Comparative Example 1 is a method for estimating the estimated value Q of the movement trajectory using a neural network, and treats P' = Q. Then, the inertial information M of smartphone B acquired in each of the three environments in the above dataset is used to plot a movement trajectory using the neural network described in Comparative Example 1. Also, the model of the neural network in Comparative Example 1 uses part of the University A environment as training data.

Comparative Example 2 is the latest Neural Inertial Localization, and corrects the movement trajectory Q from Comparative Example 1 above using a transformer network. As described above, Comparative Example 2 assumes a situation in which the environment is the same for learning and testing, and obtains a correction network model by learning different models independently for the above three environments.

Next, for each test sample used as test data, the time-unknown visited position X and the time-attached visited position _Xtimed constituting the floor plan information A were extracted, and the obstacle layout Ωobs was set. Here, the movable area in the environment was thinned, and a set of places that the user can visit (maximum 12 points from University A, 10 points from University B, and 11 points from the office) was calculated in advance as the time-unknown visited position X by detecting corners. Then, for each test sample, a place that existed at a certain distance from the true movement trajectory was used as X. In addition, the start point and end point of the true movement trajectory of each test sample were used as the time-attached visited position _Xtimed . The obstacle layout Ωobs was set by generating a two-dimensional occupancy map of the environment using the method described in the above non-patent document 1, and then manually dividing the obstacle area into a set of rectangles.

In the method of the embodiment, the estimated value Q of the trajectory is corrected based on the floor plan information A from the results of the comparative example 1, as in the comparative example 2. In the above-mentioned correction network in the embodiment, the output layer of the cross attention module is set to 32 dimensions, and the hidden layer of the MLP is set to 64 dimensions and 128 dimensions. In the objective function, λ _vp = 1.0, λ _obs = 0.01, λ _bd = 1.0, λ _reg = 0.01, and T = 100.0 are empirically set, and 2000 steps of optimization are performed with Adam with a learning rate of 0.001.

Figures 2, 3, and 4 respectively show the experimental results for the three different environments mentioned above, namely University A, University B, and Office. In Figures 2 to 4, for each environment, a movement space 40 is shown in a rectangular frame, and floor information 40b is shown in which rectangular blocks representing obstacles 41 are placed within the movement space 40. Within the movement space 40 of each environment, the true (actual) movement trajectory (Ground Truth: hereinafter sometimes referred to as "GT70") and the movement trajectories (30, 80, and 60) estimated by the methods of Comparative Example 1, Comparative Example 2, and the embodiment are shown by thick solid lines.

As shown in Figures 2 to 4, the movement trajectory 30 estimated by Comparative Example 1 deviates significantly from GT70, for example, showing movement outside the movement space 40. In the method of Comparative Example 2, in which the movement trajectory 30 estimated by the method of Comparative Example 1 is corrected, a movement trajectory 80 closer to GT70 than in Comparative Example 1 is shown, since a correction network is trained for each environment. However, it is easy to imagine that the movement trajectory 80 when the correction network of Comparative Example 2 is applied to an environment other than the environment at the time of learning will naturally deviate from GT70.

On the other hand, in the method of the embodiment, the network for trajectory estimation is not trained from an existing data set, but the movement trajectory 30 of the comparative example 1 is corrected based on the floor information 40b describing the movement space 40 and the obstacles 41 in the movement space 40, the visit location 42 with an unknown time, and the floor plan 40a including the visit location 43 with a time. In the experimental results corresponding to the embodiment in Figures 2 to 4, a part of the visit location 42 with an unknown time is shown by a white dot, and the visit location 43 with a time is shown by a black dot. And the movement trajectory 60 by the method of the embodiment is very close to the GT70. In this way, in the movement line estimation method of the embodiment, it is possible to obtain a movement trajectory P (movement trajectory 60) closer to the GT70 by setting the floor plan 40a, without learning the movement trajectory obtained by some conventional movement line estimation method (for example, the comparative example 1) that obtains the movement trajectory Q (movement trajectory 30) from the inertial information M.

In the above experiment, in order to compare the movement trajectory by the flow line estimation method according to the embodiment with the movement trajectory by other methods, data of the inertial information M obtained in the past experiment was used, and the floor plan information A was set from the above three environments in the past experiment and the true movement trajectory GT in those environments, but in a real environment, the floor plan information A is set based on the size of the real environment, the arrangement of obstacles, and the visit position (X, X _timed ) with unknown time and time attached that can be obtained in the real environment. Examples of real environments include the above-mentioned stores and facilities where traveling exhibitions and businesses are held.

Figure 5 shows the procedure for estimating a user's movement trajectory P within a store using a traffic line estimation method according to the embodiment. As shown in Figure 5, inertial information M is acquired by an IMU 10 mounted on a user terminal 1 carried by the user (s1), and the inertial information M is applied to existing inertial position estimation technology to estimate an (inaccurate) movement trajectory Q (s3).

On the other hand, since the POS system 2 can obtain the items (purchase history) of the products purchased by the user at the store, by including in the floor information 40b the correspondence between the product shelf position and the product, and the location of the checkout location (for example, the location of the POS terminal such as a cash register), the system obtains a visit location X (42) with an unknown time based on the purchase history and the floor information 40b, and a visit location X _timed with a time based on the time of checkout and the location of the checkout location included in the floor information 40b (s4). The system also obtains the position Ω _obs of an obstacle 41 in the store based on the floor information 40b (s5), and sets floor plan information A (40a) based on the obtained obstacle position Ω _obs and the visit location (X, X _timed ) (s6). Then, a calculation is performed to optimize a network that corrects an inaccurate movement trajectory Q estimated by an existing inertial position estimation technology based on the floor plan information A (40a) (s7). Then, as a result of the calculation, a more accurate movement trajectory P is output (s8).

In this way, the flow line estimation method according to the embodiment is significantly different from the map matching described in the above-mentioned Patent Document 2 in that a visited position X with an unknown time and a visited position X _timed with a time are acquired. Then, the flow line estimation method according to the embodiment makes it possible to estimate an extremely accurate flow line based on the visited position X with an unknown time and the visited position X _timed with a time, and the floor plan information A.
== ...
The flow of information processing when the traffic line estimation method according to the above embodiment is applied to a store will be specifically described below. Here, it is assumed that dedicated application software (hereinafter sometimes referred to as a "dedicated app") is installed in the user terminal 1 for collecting information required to estimate the movement trajectory P in the store and for data communication with a computer for the POS system or a computer that outputs the above inaccurate movement trajectory Q. Note that the process of finally estimating an accurate movement trajectory P based on the inertial information M may be performed by the dedicated app, but in this application example, it is performed by a computer (hereinafter sometimes referred to as a "server device") managed by the store's operating company. The server device is configured to be able to communicate with the user terminal 1 and the POS terminal in the store via a communication network such as the Internet or a LAN.

Figure 6 shows an example of a communication procedure in a computer system (hereinafter sometimes referred to as a "traffic line estimation system") consisting of a user terminal 1, a POS terminal 21, a server device 3, and a barcode reader 22. The server device 3 can be composed of a computer constituting the POS system 2, or a computer that links with the POS system 2. Of course, the server device 3 may also be a dedicated computer that executes the process of estimating the movement trajectory P while linking with the POS system 2. Here, for ease of explanation, it is assumed that the functions of the server device 3 are incorporated into the POS system 2.

The dedicated app has a function to record the inertial information M measured by the IMU 10 of the user terminal 1 in the store, a function to transmit the inertial information M to the server device 3 in a timely manner, and the like. These functions are integrated into an app (for example, the app described in the above non-patent documents 5 and 6) for using the service described in the above non-patent document 4 (hereinafter, sometimes referred to as the "automatic settlement service").

The user terminal 1 running the dedicated app timely executes the information processing and data communication required for the server device 3 to estimate the movement trajectory P while the user uses the above-mentioned automatic checkout service in the store. The server device 3 executes processing to estimate the movement trajectory P of the user in the store while providing the automatic checkout service to the user via the user terminal 1 based on data communication with the user terminal 1 and the POS terminals that make up the POS system 2. As described above, the server device 3 also serves as the computer that makes up the POS system 2. Below, with reference to FIG. 6, the information processing in the user terminal 1, POS terminal 21, and server device 3 in this application example, and the contents of data transmitted and received between the various information processing devices (1, 3, 21, 22) that make up the movement line estimation system will be described.

First, the user terminal 1 starts a dedicated app at a location that is the starting point of the movement trajectory, such as the entrance to a floor of a store (s11). When the dedicated app is started, the user terminal 1 starts recording the inertial information M (s12). Note that, in order to have the user terminal 1 start recording the inertial information M at the starting point, a user who has visited the starting point must operate the user terminal 1 and start the dedicated app. Here, as with the service described in Non-Patent Document 4 above, the user uses a shopping cart equipped with a smartphone holder. The shopping cart is kept waiting at a designated position on each floor of the store, and is used only on that floor. In other words, on other floors, similar shopping carts dedicated to that floor are also waiting.

When a user attaches their own smartphone or a dedicated smartphone always available at the store to the shopping cart as a user terminal 1 at the waiting position of the shopping cart, the user launches a dedicated app and performs an operation to start shopping. When the user uses their own user terminal 1, they can input information for specifying the movement space, such as the store or floor, into the dedicated app before starting shopping. If the smartphone is dedicated to the store, information such as the store and floor can be input in advance. The dedicated app then starts recording the inertia information M from the point in time when the operation to start shopping is accepted.

The user then moves around the store, picking out the products they wish to purchase from the shelves, having the user terminal 1 scan the barcode displayed on the product, and placing the product in their shopping cart. Each time the user terminal 1 scans a product's barcode, it records information to identify the product (hereinafter sometimes referred to as "product information") (s13).

When the user has placed all of the items they wish to purchase in the shopping cart, they proceed to a booth (cash register area) where multiple POS terminals 21 for automated checkout, known as "self-checkouts," are installed to make the payment, and have the user terminal 1 read the two-dimensional barcode 23 posted at the entrance to the cash register area (s14). The user terminal 1 then transmits the code written on the read barcode, terminal identification information for identifying the user terminal 1, such as the serial number of the dedicated app, and the product information recorded in the dedicated app, to the server device 3 via a communication network such as the Internet (s15).

When the server device 3 receives the various information transmitted from the user terminal 1, it identifies the floor information 40b of the store where the specified cash register area is located based on the received code, and obtains the floor information from an external storage device attached to the server device 3 or a database installed on the communication network (s22). The server device 3 also uses its function as the POS system 2 to identify an available POS terminal 21 installed in the cash register area, and transmits the number of the POS terminal 21 to the user terminal 1 (s23). The server device 3 also transmits payment information based on the transmitted product information (product, quantity, unit price, total amount, etc.) to the POS terminal 21 with the above number (s24). The POS terminal 21 displays the transmitted payment information, such as the product purchased by the user, its quantity, unit price, and total amount, on its own display (s31). Meanwhile, the number of the POS terminal 21 sent from the server device 3 is displayed on the user terminal 1 (s16), and the user performs the payment operation at the POS terminal 21 with that number.

When a user operates the POS terminal 21 and pays for the purchase price of a product by credit card, electronic money, or cash, the POS terminal 21 executes the settlement process (s32) and sends a notification of this to the server device 3. The server device 3 generates a two-dimensional barcode for the user terminal 1 that has completed the settlement (s25) and sends this barcode to the user terminal 1. The user terminal 1 displays the sent barcode (s17), and the user has the barcode read by the barcode reader 22 at the exit of the cash register area (s41). The barcode reader 22 immediately sends the code written in the read barcode to the server device 3, and the server device 3 transfers a confirmation of the reading of this barcode to the user terminal 1. When the user terminal 1 receives this information, it stops recording the inertial information M (s18) and sends its own terminal information and the inertial information M recorded up to that point to the server device 3 (s19).

The server device 3 obtains the time when the user terminal 1 received the read confirmation from the barcode reader 22 as the time when the user terminal 1 visited the exit of the cash register area (s26), and sets a floor plan based on the obtained time, the previously obtained floor information 40b, and the payment information (s27). The server device 3 outputs a movement trajectory P corrected based on the floor plan set by the movement trajectory Q estimated based on the inertial information M from the user terminal 1 (s29, s30). That is, two pieces of position information X timed with time are generated based on the recording start time of the obtained inertial information M, the reception time of the code transmitted from the barcode reader 22, the position X of the waiting place of the shopping cart in the specified floor information 40b, and the position X of _the exit of the cash register area, and obtains the position information X with unknown time based on the product information included in the payment information and the position of the product shelf where each product is placed in the floor information 40b. Then, the calculation for estimating the movement trajectory P based on the flow line estimation method according to the embodiment described above is performed using the position information (X, X _timed ), the inertial information M, and the floor plan information A, thereby estimating a more accurate movement trajectory P of the user. The estimated movement trajectory P may be output as appropriate, for example, by storing it in an external storage device attached to the server device 3 or a database arranged on the network, or by displaying and outputting it as a flow line as shown in Figures 2 to 4. This allows the store operating company to use the output movement trajectory P as a material for future business development and business operation, a material for evaluating advertising effectiveness, etc.
===Other Examples===
Although the embodiments and application examples of the present invention have been described above, the embodiments and application examples of the present invention are not limited to those described above, and include various modified examples and application examples. Furthermore, the above-mentioned embodiments and application examples are intended to easily explain the configuration and information processing content of the traffic line estimation method and traffic line estimation system according to the present invention, and are not necessarily limited to those having all of the configurations and information processing described. It is possible to add other configurations or other information processing, delete part of the configuration or part of the information processing, or replace it with other configurations or information processing. The order of one information processing and another information processing can also be reversed.

For example, in the above application example, the time at which recording of inertial information starts is not essential. It is sufficient to at least obtain the exit of the cash register area and the time of visiting that location at the time of payment.

In the above application example, the location information X _{timed with} time was obtained based on the location of the exit of the cash register area specified based on the floor information 40b and the visit time, but the barcode displayed by the user terminal 1 may be read by the POS terminal 21 where the user operates to settle the account. The floor information 40b may include the location of each POS terminal 21. This makes it possible to obtain the location information X _{timed with time based on the location of the POS terminal 21 where the user operates to settle the account and the time when the barcode was read by the POS terminal 21. In any case, the floor information 40b may include the location of the place where the user performs a predetermined} operation on the POS system 2 when settling the account, such as the location of the exit of the cash register area or the location of the POS terminal 21.

As described above, in the above application example, all or part of the information processing for estimating the movement trajectory P may be executed on the user terminal 1 side. The user terminal 1 and the POS terminal 21 may be configured to be able to communicate with each other. When the information processing for estimating the movement trajectory P is executed on the user terminal 1 side, the latest floor information 40b may be downloaded from the server device 3 at the time of visiting the starting point or the end point of the movement trajectory P. Of course, not limited to the above application example, even in the above-mentioned traveling exhibition and business, the user terminal carried by the user may execute all or part of the information processing for estimating the movement trajectory. In any case, it is sufficient that the computer system for estimating the movement trajectory P is composed of one or more information processing devices including at least the user terminal 1.

In the above application example, if a store has multiple entrances, or if a store has multiple floors and purchases made on one floor can be paid for on another floor, or if payment can be made on a different floor and each floor has multiple entrances, the user may be required to input the entrance, floor number, or even a combination of a floor number and the entrance to that floor when starting up the dedicated app.

A QR code (registered trademark) or the like containing information about the starting point location may be posted at the entrance, and by reading the code with the user terminal 1, a dedicated app may be launched and floor information 40b may be identified. If the dedicated app is set to run constantly in the background, a beacon transmitter may be installed at the starting point location. Then, when the user terminal 1 receives a beacon, it stores the beacon identifier using the function of the running dedicated app, and starts recording the inertial information with the IMU 10. In any case, it is sufficient to be able to identify the starting point location and the time when recording of the inertial information, which is the time that marks the starting point of the movement trajectory.

The functions of the dedicated app may be integrated into an app for using well-known point services or coupon services provided by a store operating company, etc. In such a case, the user terminal displays a barcode containing the serial number of the dedicated app, etc., and has the POS terminal 21 read the barcode at the time of payment. The POS terminal 21 then transmits the read code, payment information, and its own identification information to the server device 3, and the user terminal 1 transmits terminal identification information and inertia information M to the server device 3 at the time of paying for the product to the POS terminal 21 using an electronic payment service, allowing the server device 3 to obtain the information necessary to estimate the movement trajectory P.

When the flow line estimation method according to the embodiment is applied to the above-mentioned patrol inspection work in a facility such as a factory, the patrol inspection work can specify the inspection location as a visit location X with an unknown time by including the location of the inspection location corresponding to the inspection list in the target floor information 40b. In addition, the start and end locations of the inspection are often specified. In this case, the inspector can specify the visit location X _timed with the time attached by inputting the start and end times of the inspection into the server device 3 via the user terminal 1 after the inspection. This allows the floor plan 40a to be set. The start and end times and locations of the patrol inspection may be determined. Alternatively, the inspector can carry out the patrol inspection work with the user terminal 1 and specify the visit location X _timed with the time attached by inputting the inspection confirmation or the start or end of the inspection into the user terminal 1 at a specified inspection location or at the start and end of the inspection.

The position X _timed accompanied with the time acquired in the position estimation method according to the above embodiment is not limited to being based on floor information such as the waiting position of the cart or the position of the POS terminal in the above application example, but may be based on time and position information obtained by a sensing technology other than the IMU 10, such as an AI camera or a beacon, or a combination of multiple sensing technologies other than the IMU 10.

When combining an AI camera with the traffic line estimation method according to the embodiment, for example, the positions and movement directions of an unspecified number of users detected by the AI camera from images captured by a security camera or the like can be collated with the traffic lines of users of the dedicated app estimated by the method of the above embodiment, thereby making it possible to identify users of the dedicated app included in the images captured by the camera. Then, a time-stamped position X _timed can be obtained based on the position and time when the identified users were photographed.

When beacons are combined with the traffic flow estimation method according to the embodiment, in recent years, retail stores and the like have been using BLE standard beacons to obtain user location information for purchase promotion, etc., and by measuring the signal strength of a beacon emitted from a transmitter installed in the store with the user terminal 1, a combination of time and approximate location can be obtained.

In addition, information obtained using a sensing technology other than an IMU may include ambiguity in the visited position x _k , such as the shooting range of an AI camera or the receiving range of a beacon, but in such a case, the visited position X _timed with time may be obtained by treating the visited position not as a point but as a set of coordinates with a spread. In any case, the flow line estimation method according to the embodiment may complement a conventional flow line estimation method using an AI camera, a beacon, or the like.

In the flow line estimation method according to the above embodiment, the movement trajectory Q to be corrected is obtained from the inertial information M based on the method of Comparative Example 1, but as is well known, in the field of robotics and the like, there are various methods for estimating the movement trajectory Q from the inertial information M. The essence of the flow line estimation method according to the embodiment is to obtain the movement trajectory Q from the inertial information M by an existing inertial position estimation method, and then correct the movement trajectory Q based on floor information 40b and floor plan information A (40a) including the visited position (X, X _timed ).

1 User terminal, 2 POS system, 3 Server device, 10 IMU,
20 Neural network, 21 POS terminal, 22 Barcode reader, 23 Two-dimensional barcode, 30, 60, 70, 80 Movement trajectory, 40 Movement space,
40a Floor plan (floor plan information), 40b Floor information, 41 Obstacles,
42, 43 Visiting location, 50 Correction network

Claims (7)

A method for estimating a user's movement line in a moving space using a computer system, comprising:
The computer system comprises:
a trajectory generating step of generating a first movement trajectory based on inertial information outputted from a user terminal equipped with an inertial measurement device and moving in the movement space together with the user;
a floor information acquisition step of acquiring floor information describing a layout of the movement space;
a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space;
a floor plan setting step of setting a floor plan including the floor information and the visiting location;
a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory;
Run
In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired,
In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position.
Method for estimating traffic flow. The moving space is a store,
The computer system is linked to a POS system of the store;
A product information acquisition step of acquiring information on a product purchased by a user via the POS system,
The floor information includes the locations of product shelves in the store, information on products displayed on each product shelf, and the location of a place where a predetermined operation for a POS system is performed when making a payment,
In the visiting location acquisition step, the visiting location with unknown time based on the product information acquired in the product information acquisition step and the visiting location with time based on a predetermined operation time for the settlement are acquired.
The method for estimating a traffic line according to claim 1 . The floor information includes a recording start position at which recording of the inertial information is started by the user terminal,
acquiring the visited location with a time stamp based on a time when an instruction to start recording the inertial information for the user terminal was received and the recording start location;
The method for estimating a traffic line according to claim 2 . The moving space is a facility where a patrol inspection is performed,
The floor information includes an inspection location in the facility,
In the user position acquisition step, the inspection position is acquired as the visit position, and the visit position includes the visit position accompanied with a time based on a user input to a predetermined device at the predetermined inspection position.
The method for estimating a traffic line according to claim 1 . The movement line estimation method according to claim 1, wherein in the trajectory generation step, a trained neural network estimates the first movement trajectory. The traffic line estimation method according to any one of claims 1 to 5, wherein the visited locations with associated time include those obtained based on at least one of an image captured by an AI camera and a beacon received by the user terminal. A computer system that is configured with one or more information processing devices and estimates a user's movement line in a moving space,
The information processing device includes a user terminal equipped with an inertial measurement device and moving in the moving space together with the user;
storing floor information describing the layout of the travel space;
a trajectory generating step of generating a first movement trajectory based on inertial information output by the user terminal;
a visiting position acquiring step of acquiring visiting positions where the user has stopped in the moving space;
a floor plan setting step of setting a floor plan including the floor information and the visiting location;
a trajectory correction step of correcting the first movement trajectory based on the floor plan to output a second movement trajectory;
Run
In the visiting position acquisition step, the visiting position with an unknown time and the visiting position with a time are acquired,
In the trajectory correction step, the first movement trajectory is corrected based on a movable area in the movement space included in the floor plan, a position of an obstacle that cannot be moved, and the visited position.
Traffic flow estimation system.