JP2024036380A - Information processing device, information processing method, and program - Google Patents

Abstract

To conduct management of articles related to entry into and exit from a shelf by using an imaging part.SOLUTION: An information processing device that communicates to an imaging part to achieve the purpose mentioned above includes: moving direction identifying means that identifies the moving direction of an article based on an image taken of the front direction of the shelf by the imaging part; entry and exit identifying means that identifies the entry or exit of the article by the direction identified by the moving direction detection means; managing means that manages the number related to the articles on the shelf based on the entry and exit identified by the identifying means.SELECTED DRAWING: Figure 1

Description

The present invention relates to an information processing device, an information processing method, and a program that manage shelf inventory using an imaging unit.

In recent years, in the logistics industry such as retail and distribution, there has been a need for more precise and efficient inventory management than ever before due to a shortage of human resources and to improve logistics efficiency. In particular, counting the current inventory on shelves where product inventory is stored is still done manually.

RFID is known as a means to solve these problems. RFID is a system in which an IC chip called an RF tag is attached to each product in stock, and the product inventory is counted through wireless communication.

However, in actual operation, physical and human costs are incurred to attach RF tags to each product, and the current situation is that cost performance commensurate with operation cannot be obtained in most retail and logistics sites. It is.

Also known is a technique for counting the number of products by attaching weight sensors to shelves. However, there are problems in that the cost of installing a weight sensor on each shelf is not worth it, and that accurate inventory counts cannot be made due to variations in the weight of products.

Under these circumstances, a technology has been devised that uses computer vision (image processing) technology to manage shelf inventory. US Pat. No. 5,080,001 discloses a system for tracking the removal or placement of items in an inventory location with material handling facilities.

Special table 2016-532932

Patent Document 1 describes a technology for determining which product has been taken from a shelf by detecting the position of the customer's hand from an image when the customer takes the product from the shelf.

However, in the case of the above technology, an imaging device that images the inventory location or a presence detection device that detects the inventory status is required. Since an imaging device or a presence detection device must be arranged for each shelf or row of shelves, there is a problem in that the cost increases.

Therefore, an object of the present invention is to use an imaging unit to manage articles that are taken in and out of shelves.

In order to achieve the above object, the present invention provides an information processing device capable of communicating with an imaging unit, the information processing device specifying a direction of movement of an article based on an image taken in the front direction of a shelf by the imaging unit. a specifying means; an entry/exit specifying means for specifying the exit or entry of the article based on the direction specified by the movement direction specifying means; a position of the article detected by the article position detection section; It is characterized by comprising a management means for managing the number of articles on the shelf based on the arrival and departure.

According to the present invention, it is possible to manage articles that are taken in and out of shelves using the imaging unit.

1 is a configuration diagram schematically showing an inventory management system according to an embodiment of the present invention. 1 is a configuration diagram showing the hardware configuration of an information processing device 102 and various servers according to an embodiment of the present invention. FIG. 1 is a configuration diagram showing a hardware configuration of a network camera 101 according to an embodiment of the present invention. FIG. 2 is a flowchart illustrating an overview of inventory management processing in the information processing apparatus 102 according to the embodiment of the present invention. 2 is a flowchart showing an overview of product inventory tracking processing in the information processing device 102 according to the embodiment of the present invention. 2 is a flowchart illustrating an overview of product retrieval processing in the information processing device 102 according to the embodiment of the present invention. 2 is a flowchart showing an overview of product return processing in the information processing device 102 according to the embodiment of the present invention. FIG. 2 is a schematic diagram illustrating an image of a product shelf and a region (virtual grid region) from which products are taken out in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image at the time of starting to take out products in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image during product removal in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image at the end of product removal in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image at the time of starting product return in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image during product return in an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating a product shelf and a processing image at the end of product return in an embodiment of the present invention. FIG. 3 is a schematic diagram illustrating an image of detecting another type of product within a product extraction area (virtual grid area) according to an embodiment of the present invention. FIG. 3 is a schematic diagram illustrating an image of detecting another product of the same type within an area (virtual grid area) from which a product is taken out according to an embodiment of the present invention. FIG. 2 is a schematic diagram illustrating an image of a product shelf with a height direction and an area (virtual grid area) from which products are taken out in an embodiment of the present invention. This is an example of data, which is included in the information processing apparatus according to the embodiment of the present invention, and is used to identify a shelf where a product is entered or removed from the warehouse based on the relationship between the coordinates of the virtual grid where the product is detected and the depth. It is an example of the data table which stores the inventory quantity of the product shelf in embodiment of this invention. 12 is a flowchart illustrating a processing overview of a second embodiment of inventory management in the information processing apparatus 102 according to the embodiment of the present invention. It is a flowchart which shows the processing outline of 2nd Embodiment of product inventory tracking in the information processing apparatus 102 based on embodiment of this invention. FIG. 2 is a schematic diagram illustrating an image of a product shelf having a plurality of shelves in the height direction and an area (detection area) from which products are taken out in an embodiment of the present invention. It is an example of a series of product images stored in the information processing device 102 according to the embodiment of the present invention.

<First embodiment>
Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a configuration diagram showing an outline of an inventory management system according to an embodiment of the present invention.

The inventory monitoring system 105 is connected to a network camera 101 and an information processing device 102 that processes images captured by the camera, and the network camera 101 images the outlet of the product shelf 103 from top to bottom. Products 104 are displayed on the product shelf 103. Note that although the network camera 101 is described as an example in which the image is taken from the top to the bottom of the ejection port, the image of the ejection port may be imaged from the side or diagonally from above. In the present embodiment, an example will be described in which images are taken from the top to the bottom of the outlet.

The status of the products monitored by the inventory monitoring system 105 is counted via the network 106, for example, by an inventory management server 107 on a cloud, and the inventory status is provided to the user. Note that although the inventory management server 107 and the information processing device 102 are illustrated as being in separate casings in FIG. 1, they may each be in one casing. That is, the information processing device 102 may have the function of the inventory management server 107, or the inventory management server 107 may have the function of processing captured images possessed by the information processing device 102.

The information processing device 102 may acquire information from one network camera 101 and send it to the inventory management server 107 as shown in FIG. Information may be collected and sent to the inventory management server 107.

Next, a schematic hardware configuration of the information processing apparatus 102 will be described using FIG. 2.

FIG. 2 is a configuration diagram showing the hardware configuration of the information processing device 102 and various servers according to the embodiment of the present invention.

The CPU 201 centrally controls each device and controller connected to the system bus 204.

In addition, the ROM 202 or external memory 211 includes BIOS (Basic Input/Output System), which is a control program for the CPU 201, an operating system program (hereinafter referred to as OS), and information necessary to realize the functions executed by the image processing server 108. Various programs, etc., which will be described later, are stored. The RAM 203 functions as the main memory, work area, etc. of the CPU 201.

The CPU 201 loads programs and the like necessary for execution of processing into the RAM 203 and executes the programs to realize various operations.

Further, an input controller (input C) 205 controls input from a keyboard as an input unit 209 and a pointing device such as a mouse (not shown).

A video controller (VC) 206 controls display on a display device such as a CRT display (CRT) serving as the display unit 210. The display device may be not only a CRT but also a liquid crystal display. These are used by the administrator as needed. It is not directly related to the present invention.

A memory controller (MC) 207 is a hard disk (HD), floppy disk (registered trademark FD), or PCMCIA card slot that stores boot programs, browser software, various applications, font data, user files, editing files, various data, etc. Controls access to an external memory 211 such as a CompactFlash (registered trademark) memory connected to the PC via an adapter.

A communication I/F controller (communication I/FC) 208 connects and communicates with external devices via a network, and executes communication control processing on the network. For example, Internet communication using TCP/IP is possible. It also has the function of a communication I/F controller that can be connected to the network camera 101 via a network.

Note that the CPU 201 enables display on the display unit 210 by, for example, executing an outline font development (rasterization) process in a display information area in the RAM 203. Further, the CPU 201 allows the user to give instructions using a mouse cursor (not shown) on the display unit 210.

A program for realizing the present invention is recorded in the external memory 211, and is executed by the CPU 201 by being loaded into the RAM 203 as necessary. Further, definition files and various information tables used by the program according to the present invention are stored in the external memory 211, and detailed explanations thereof will be given later.

Next, the schematic hardware configuration of the network camera 101 will be described using FIG. 3.

FIG. 3 is a configuration diagram showing the hardware configuration of the network camera 101.

The CPU 301 centrally controls each device and controller connected to the system bus 304.

In addition, the ROM 302 includes BIOS (Basic Input/Output System), which is a control program for the CPU 301, an operating system program (hereinafter referred to as OS), and the functions necessary to generate data to be sent to the information processing device 102. Various programs etc. are stored. The RAM 303 functions as the main memory, work area, etc. of the CPU 301.

The CPU 301 loads programs and the like necessary for execution of processing into the RAM 303 and executes the programs to realize various operations.

The RGB camera unit 307 is connected to an image processing unit 308, and after photoelectrically converting the light obtained by passing through a lens directed toward the monitoring target using a light receiving cell such as a CCD or CMOS, the RGB camera unit 307 converts the light into RGB images. The signal and complementary color signal are output to the image processing unit 308.

The image processing unit 308 performs white balance adjustment, gamma processing, and sharpness processing based on the RGB signal and color capture signal, and further performs YC signal processing to generate a luminance signal Y and a chroma signal (hereinafter referred to as YC signal). Then, the YC signal is compressed in a predetermined compression format (for example, JPEG format or Motion JPEG format), and this compressed data is temporarily stored in RAM 303 as image data.

The TOF sensor 305 is an image sensor that measures the distance to an object using the TOF (Time-of-Flight) method. By measuring together with the measurement unit 306, the distance to the subject is measured. Note that as a means of measuring the depth of the product, the depth may be measured using a stereo camera configured with two RGB cameras 307, or the depth may be estimated from the results of two-dimensional image analysis using image processing or deep learning. You can also use a method.

A communication I/F controller (communication I/FC) 309 connects and communicates with external devices via a network, and executes communication control processing on the network, and image data stored in the RAM 303 is , is transmitted by the communication I/F controller 309 to the information processing device 102, which is an external device.

Next, the flow of inventory management processing according to the first embodiment of the present invention will be explained with reference to FIGS. 4 to 7.

FIG. 4 is a flowchart showing an overview of inventory management processing in the information processing apparatus 102 according to the embodiment of the present invention, and S401 to S407 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 4 is a flow of processing that is started when the network camera 101 and information processing device 102 are activated. Note that in the subsequent flowcharts, it is assumed that the information processing apparatus 102 can acquire data from the network camera 101 through the network at any time.

Before starting the process of the flowchart in FIG. 4, the CPU 201 of the information processing apparatus 102 first obtains a captured image of the outlet of the product shelf obtained by the RGB camera 307 of the network camera 101.

Next, in step S401, the CPU 201 of the information processing device 102 sets a virtual grid that matches the lanes of the product shelf based on the acquired captured image. An example of virtual grid setting will be described with reference to FIGS. 8, 17, and 18.

FIG. 8 is a schematic diagram illustrating an image of a product shelf and a region (virtual grid region) from which products are taken out in the embodiment of the present invention.

In FIG. 8, 103 is an example of a product shelf, 104 is an example of a product, and 404, which is an imaging range, corresponds to a product outlet. Although FIGS. 8 to 17 will be described below with an image of the outlet from the product shelf being imaged from above, it is also possible to image the outlet from the product shelf being imaged from the side. In that case, the coordinates and depth of the product will be swapped in FIG. 18, which will be described later.

The product shelf 103 in FIG. 8 has lanes 410 arranged for each product, and positions 405 and 406 that serve as boundaries between the lanes.

Boundaries obtained by extending the boundary positions 405 and 406 to the imaging range 404 are set as 407 and 408, and a virtual grid 401 is created in the imaging range 404 to separate them. That is, a virtual grid is set as a take-out lane for each lane of each product shelf.

FIG. 17 shows the results of performing these processes on multiple shelves.

FIG. 17 is a schematic diagram illustrating an image of a product shelf with a height direction and an area (virtual grid area) from which products are taken out in the embodiment of the present invention.

The product shelf 103 in FIG. 17 is composed of multiple shelves, and FIG. 17 shows an example in which there are three shelves.

The virtual grid of the three-tier shelf in FIG. 17 differs depending on the number of shelves; the border of the first product lane is 1204, the border of the second lane is 1205, and the border of the third lane is 1206. The boundaries of the virtual grid in each stage are set as 1207 for the first stage, 1208 for the second stage, and 1209 for the third stage.

A virtual grid of three shelves is set as shown in FIG. 17, and data relating the positions of the respective shelves will be explained with reference to FIG.

FIG. 18 is an example of data, which is included in the information processing apparatus according to the embodiment of the present invention, and is used to identify a shelf from which a product enters or exits based on the relationship between the coordinates of the virtual grid where the product is detected and the depth.

1800 in FIG. 18 is data that specifies the position of the product shelf from which the product is entered and exited from the depth 1801 obtained from the depth measurement unit 306 and the coordinates 1802 of the product position obtained by the RGB camera. If the grid coordinates (horizontal coordinates in FIG. 17, with the center as the origin) are at position 70 for the product, it is determined that the product has entered or exited from grid C in the first row (that is, grid C-1). Further, if a product enters and exits at a depth of 80 cm and the grid coordinates are at a position of -60, it is determined that the product entered and exited from the third row of grid A (that is, grid A-3). Note that the direction of entry and exit will be described later in the next step S402.

As shown in FIG. 18, it has a table that stores the positions of shelves where products go in and out. This table may be manually input by the user, or may be automatically set from the boundaries (partitions) of the product shelves in the image captured before the process in step S401. Returning to the explanation of the flowchart in FIG. 4.

In the next step S402 in FIG. 4, the CPU 201 of the information processing device 102 sets the direction in which the product is being taken out based on the direction of the product moving within the virtual grid. This will be explained with reference to FIG.

In FIG. 8, the position of the product shelf 103 is at the bottom of the drawing, and the outlet side (virtual grid side) 404 is at the top of the drawing, so the direction of product removal is positive in the Y-axis direction of the coordinate axis 420. The upward direction is set as in . This removal direction may be set manually by the user, or may be set by specifying the position of the product shelf 103 on the imaging screen (FIG. 8). Returning to the explanation of the flowchart in FIG. 4.

Next, in step S403 of FIG. 4, the CPU 201 of the information processing apparatus 102 receives an input of the number of stocks in the lane of the product shelf corresponding to the virtual grid set in step S401. This will be explained with reference to FIGS. 8 and 19.

When the virtual grids in Figure 8 are set, the stock in the lane of the product shelf corresponding to each virtual grid is, for example, 10 items (411) for grid A, 5 items (412) for grid B, and 1 item for grid C. (413), the user inputs the inventory quantity from the input unit 209 of the information processing device 102 and sets the inventory quantity for each lane. An example of data for storing the set inventory quantity will be explained with reference to FIG. 19.

FIG. 19 is an example of a data table that stores the number of items in stock on the product shelf according to the embodiment of the present invention.

In the table of FIG. 19, the inventory quantities in the first row of FIG. 8 are set, and data of 10 items in grid A, 5 items in grid B, and 1 item in grid C are registered, respectively. Note that the table in FIG. 19 may be stored in the information processing device 102 or may be stored in the inventory management server 107. Returning to the explanation of the flowchart in FIG. 4.

In the next step S404 in FIG. 4, the CPU 201 of the information processing apparatus 102 transmits a command to start imaging (image capture) to the network camera 101, and starts imaging.

In the next step S405, the CPU 201 of the information processing apparatus 102 determines whether a product is detected within the imaged virtual grid 401. For example, when a network camera is installed on the top as shown in Figure 8, the product is detected by storing the shape of the product from above and using the SIFT (Scale-Invariant Feature Transform) algorithm to image the product. It may also be detected by matching. Alternatively, various images taken from above of the product may be registered in advance as learning data and subjected to machine learning, and the product may be detected using image recognition AI.

In step S405, if a product is detected, the process moves to step S406, and if no product is detected, the process moves to step S407.

When the process moves to step S406, the CPU 201 of the information processing apparatus 102 performs product tracking processing to understand the inventory status of the product detected in step S405. The processing contents of step S406 will be explained with reference to FIG.

FIG. 5 is a flowchart showing an overview of product inventory tracking processing in the information processing apparatus 102 according to the embodiment of the present invention, and S501 to S514 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 5 is a flow of processing that is started when the flowchart in FIG. 4 transitions to step S406.

First, in step S501 of FIG. 5, the CPU 201 of the information processing apparatus 102 acquires the location (coordinates) of the detected product from the captured image (capture) acquired from the network camera 101, and stores it as the start coordinate. A description with reference to specific product images will be given later with reference to FIGS. 9 to 14.

Next, in step S502, the CPU 201 of the information processing apparatus 102 obtains the height (depth) of the detected product using the TOF sensor 305 included in the network camera 101, and stores it as the starting depth.

Next, in step S503, the CPU 201 of the information processing apparatus 102 acquires the next captured image (capture) from the network camera 101.

Next, in step S504, the CPU 201 of the information processing apparatus 102 determines whether the same type of product as the previous one was detected within the virtual grid 401 imaged in step S503. If products of the same type are detected, the process transitions to step S505, and if products of the same type are not detected, the process transitions to step S509. Note that if a product of a different type from the previous one is detected in the virtual grid 401 imaged in step S503 (as in FIG. 15), the flowchart of FIG. 5 is executed from step S501 for the different type of product.

When the process transitions to step S505, the CPU 201 of the information processing apparatus 102 acquires the location (coordinates) of the detected product from the captured image (captured) taken in step S503, and stores it as the current coordinate. In addition, the current coordinates and product type that were obtained one time ago are also stored as the "previously obtained current coordinates" and the previous product type.

Next, in step S506, the CPU 201 of the information processing apparatus 102 acquires the height (depth) of the product detected when the image was captured in step S503, and stores it as the current depth.

Next, in step S507, the CPU 201 of the information processing apparatus 102 compares the distance between the "previously acquired current coordinates" and the newly acquired current coordinates in step S505, and if the distance is greater than or equal to a predetermined threshold, If there are only products, the process moves to step S510. On the other hand, if there is a product for which the distance between the "previously acquired current coordinates" and the newly acquired current coordinates in step S505 is within the threshold value, the process moves to step S508. A specific example will be described later with reference to FIG. 16.

The decision branch in step S507 is a decision branch that occurs when multiple products of the same type enter the virtual grid 401 at the same time, and the distance between the two products is equal to or greater than the threshold, that is, each individual product is different. This means that it is determined that there is. If the distance between the two is within the threshold, the product at the "previously acquired current coordinates" and the product newly detected in step S505 are recognized as the same individual (processing in step S508).

After the process in step S508, the process returns to step S503, and the process is repeated from the process of acquiring the captured image within the virtual grid 401.

On the other hand, in step S504, if the same type of product as last time is not detected in the virtual grid 401, the process moves to step S509.

In step S509, the CPU 201 of the information processing device 102 determines whether the same product has not been detected a predetermined number of times or more. If the same product cannot be detected a predetermined number of times or more, the process moves to step S510, and if the same product can be detected within the predetermined number of times, the process returns to step S503.

The decision branch in step S509 is such that even if there is a product within the virtual grid 401 but the image captured by the imaging unit cannot temporarily detect the product due to an afterimage or light disturbance, the product may be outside the virtual grid 401. It will be a branch of judgment to avoid determining that it has appeared.

In the next step S510, either the current coordinates have been stored at least once in step S509 (YES in step S509), or the current coordinates newly detected in step S505 are stored as the "previously acquired current coordinates" in step S507. Transition occurs when the distance from the coordinates is more than a threshold value.

In step S510, the CPU 201 of the information processing device 102 uses the starting coordinates, which is the first detected position of the product obtained in step S501, and the "last current coordinates", which is the detected position of the product last obtained in step S505. , calculate the direction of movement of the product.

Next, in step S511, the CPU 201 of the information processing apparatus 102 divides processing depending on the orientation calculated in step S510.

If the starting coordinates and the "last current coordinates" have the same direction, for example, if the coordinate axis 420 in FIG. 8 has the same direction from the center of the virtual grid in the Y-axis direction, the process of the flowchart in FIG. 5 ends. This determination is made when an item enters the virtual grid but does not leave or leave it (for example, a customer takes the item off the shelf and considers purchasing it, but also stops purchasing and returns it to the shelf). There will be a split decision.

Further, in step S511, if the direction of movement of the product is the take-out direction 402, the process transitions to step S512.

On the other hand, in step S511, if the direction of movement of the product is the return direction (opposite to the direction of removal), the process transitions to step S513.

When the process transitions to step S512, the CPU 201 of the information processing device 102 performs a product removal process, and the process transitions to step S514. Detailed processing will be described later with reference to FIG.

Furthermore, when the process transitions to step S513, the CPU 201 of the information processing device 102 performs a product return process, and the process transitions to step S514. Detailed treatment will be described later with reference to FIG.

When the process transitions to step S514, the CPU 201 of the information processing apparatus 102 notifies the inventory management server to change the number of stocks on the shelf identified in step S512 or step S513, and ends the flowchart of FIG. Returning to the explanation of the flowchart in FIG. 4.

In step S406, after performing product tracking processing to grasp the inventory status of the product detected in step S405, in the next step S407, whether to end the imaging process of the network camera 101 for checking product inventory. to decide. If the imaging process is to be continued, the process returns to step S404, and the process from step S404 is repeated. On the other hand, if there is an instruction from the user to end the imaging process, the imaging process is stopped and the processing of the present invention is ended.

Next, with reference to FIG. 6, the process of identifying the position of the shelf from which the product is to be taken out will be described.

FIG. 6 is a flowchart showing an overview of the product retrieval process in the information processing apparatus 102 according to the embodiment of the present invention, and S601 to S602 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 6 is a flow of processing that is started when the process transitions to step S512 in the flowchart in FIG.

First, in step S601 of FIG. 6, the CPU 201 of the information processing apparatus 102 identifies the number of shelves taken out from the starting depth obtained in step S502. To explain specifically with reference to FIG. 18, for example, if the depth of the product acquired in step S502 is 20 cm, it is specified from the data 1801 and 1803 that the height of the retrieved shelf is the first level. .

Next, in step S602, the CPU 201 of the information processing apparatus 102 identifies the position of the extracted shelf from the start coordinates obtained in step S501. To explain specifically with reference to FIG. 18, for example, if the grid coordinates of the product acquired in step S501 are 80, then from the data 1802, C-1 where the grid coordinates of the first row are 80, that is, the first row. It is identified that the shelf is C.

In the above example, it is specified that the product was taken out from the shelf C in the first stage.

Through the above processing, it is possible to specify the position of the shelf from which the customer took out the product.

Next, with reference to FIG. 7, the process of identifying the position of the shelf where the product was returned will be described.

FIG. 7 is a flowchart showing an overview of the product retrieval process in the information processing apparatus 102 according to the embodiment of the present invention, and S701 to S702 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 7 is a flow of processing that is started when the flowchart in FIG. 5 transitions to step S513.

First, in step S701 of FIG. 7, the CPU 201 of the information processing apparatus 102 specifies the number of shelves returned from the last current depth obtained in step S506. Specifically, referring to FIG. 18, for example, if the depth of the product acquired in step S502 is 80 cm, it is determined from the data 1801 and 1803 that the returned shelf is the third shelf.

Next, in step S702, the CPU 201 of the information processing apparatus 102 identifies the position of the shelf taken out from the last current coordinates obtained in step S505. Specifically, referring to FIG. 18, for example, if the grid coordinate of the product acquired in step S501 is 80, then from the data 1802, C-3 where the grid coordinate of the third row is 80, that is, the third It is identified that the shelf is in the C position.

In the above example, it is specified that the product was returned to the third shelf C.

Through the above processing, it is possible to specify the position of the shelf where the customer returned the product.

Next, with reference to FIGS. 9 to 14, a process for identifying the position of a shelf where a product enters and exits from the shelf based on an image of the product going in and out of the shelf will be specifically described.

FIG. 9 is a schematic diagram illustrating a product shelf and a process image at the start of product removal in the embodiment of the present invention.

In FIG. 9, when the product 104 is detected as 801 in the virtual grid 401 (processing in step S405), first, the starting coordinates 802 of the product are specified (processing in step S501). Subsequently, the starting depth of the product is also specified (processing in step S502), and each value is stored.

FIG. 10 is a schematic diagram illustrating a product shelf and a processing image during product removal in the embodiment of the present invention.

In FIG. 10, when the product 104 moves within the virtual grid, the current coordinates of the current coordinates 802 to 804 are continued to be acquired until the product 104 disappears from the virtual grid (processing in step S505).

FIG. 11 is a schematic diagram illustrating a product shelf and a processing image at the end of product removal in the embodiment of the present invention.

In FIG. 11, when the current coordinates of the product 104 no longer exist in the virtual grid, such as 805 (transition to NO in the decision branch of step S504), 804 is set as the current coordinates in the last virtual grid ( (Processing in step S510).

Next, the direction of movement 806 of the product is calculated from the information on the starting coordinates 801 and the final current coordinates 804 of the product. In the case of FIG. 11, since it is the same as the take-out direction 402 in FIG. 8, it is specified as the take-out direction (processing in step S510).

If the product moves as shown in the images from FIG. 9 to FIG. 11, it is determined in step S511 that the product has moved in the direction to be taken out, and the process moves to step S512 (ie, the process in FIG. 6).

In the case of FIG. 11, the position of the shelf taken out is specified from the starting depth obtained at the same time as the starting coordinates 801 (processing in steps S601 and S602 in FIG. 6). In FIG. 11, one product has been taken out from grid B (center shelf), so the number of items in stock is reduced from 5 to 4 as shown in 807. Information for reducing the inventory quantity is transmitted to the inventory management server 107 (processing in step S514).

As shown in the image above, by capturing an image of the shelf opening, it is possible to understand the product takeout status.

Next, an image of the process when the product is returned to the shelf will be described with reference to FIGS. 12 to 14.

FIG. 12 is a schematic diagram illustrating a product shelf and a processing image at the time of starting product return in the embodiment of the present invention.

In FIG. 12, when the product 104 is first detected as 901 in the virtual grid 401 (processing in step S405), the starting coordinates 901 of the product are first identified (processing in step S501). Subsequently, the starting depth of the product is also specified (processing in step S502), and each value is stored.

FIG. 13 is a schematic diagram illustrating a product shelf and a processing image during product return in the embodiment of the present invention.

In FIG. 13, when the product 104 moves within the virtual grid, the current coordinates of current coordinates 902 to 905 are continued to be obtained (processing in step S505) until the product 104 disappears from the virtual grid. Subsequently, the current depth value also continues to be obtained.

FIG. 14 is a schematic diagram illustrating a product shelf and a processing image at the end of product return in the embodiment of the present invention.

In FIG. 14, when the current coordinates of the product 104 no longer exist in the virtual grid like 906 (transition to NO in the decision branch of step S504), 905 is set as the current coordinates in the last virtual grid ( (Processing in step S510).

Next, the direction of movement 907 of the product is calculated from the information of the start coordinates 901 and the final current coordinates 905 of the product. In the case of FIG. 14, the direction 907 of movement of the product is opposite to the take-out direction 402 of FIG. 8, so it is specified as the return direction (processing in step S510).

If the product moves as shown in the images from FIG. 12 to FIG. 14, it is determined in step S511 that the product has moved in the backward direction, and the process shifts to step S513 (ie, the process in FIG. 7).

In the case of FIG. 14, the position of the shelf taken out is specified from the starting depth obtained at the same time as the last current coordinate 905 (processing in steps S701 and S702 in FIG. 7). In the example of FIG. 14, one product has been returned to grid A (left shelf), so the number of items in stock is increased from the original 10 to 11 as shown in 908. This information to increase the inventory quantity is transmitted to the inventory management server 107 (processing in step S514).

As shown in the image above, by capturing an image of the take-out opening of the shelf, it is possible to grasp the status of returned products.

Next, an image of the processing in steps S507 and S508 will be described with reference to FIG. 16.

FIG. 16 is a schematic diagram illustrating an image of detecting another product of the same type within the virtual grid area 401 according to the embodiment of the present invention.

The image in FIG. 16 is an example where after the product 104 is detected in 1102 (step S501), the same product is detected in 1103 and 1104 in the next imaging process (step S503).

In FIG. 16, when the threshold indicating the movement of the product is 1101, the distance between the start coordinate 1102 and the current coordinate 1104 is more than the threshold (1106), and on the other hand, the current coordinate 1103 is within the threshold (1105). It is determined that the product moved from the start coordinate 1102 has moved to 1103. The threshold value is set based on the speed at which the customer moves the product and the time interval at which image processing is performed. This threshold value needs to be shorter than the lane interval of the product shelf (distance between 405 and 406 in Figure 8), so the time interval for image capture processing should be set in relation to the customer's product movement speed. It's okay.

With the above processing, when the frame rate of imaging is slow and there is a limit to the speed of tracking products, even if two products are recognized, if they are separated by a threshold value of 1101 or more, the two products can be identified and placed in and out of the shelf. can be managed.
<Second embodiment>
In the first embodiment, the product is tracked after the product is identified from the image captured by the imaging unit (network camera) 101. In the second embodiment, by performing product identification after tracking the product, it is possible to eliminate the process of performing product identification from a captured image every time an image is captured, and increase the frame rate of the imaging unit. That is, in the second embodiment, when a product leaves the detection area (virtual grid) 401, product identification processing is performed all at once from the accumulated images, thereby preventing product identification processing every time an image is captured, and reducing the processing time of the imaging unit. Increasing frame rate. The flow of inventory management processing according to the second embodiment of the present invention will be described with reference to FIGS. 20 and 21.

FIG. 20 is a flowchart showing a processing overview of the second embodiment of inventory management in the information processing apparatus 102 according to the embodiment of the present invention, and S2001 to S2003 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 20 is a flow of processing that is started when the imaging unit 101 and information processing device 102 are activated. Note that in the subsequent flowcharts, it is assumed that the information processing apparatus 102 can acquire data from the network camera 101 through the network at any time.

Before starting the process of the flowchart in FIG. 20, the CPU 201 of the information processing device 102 first obtains a captured image of the outlet portion of the product shelf obtained by the RGB camera 307 of the imaging unit 101.

Next, in step S2001, the CPU 201 of the information processing device 102 accepts settings of the detection area 401 that matches the lane and height of the product shelf based on the acquired captured image. The method of setting the detection area is the same as in FIGS. 8, 17, and 18, so the explanation will be omitted. Note that in the first embodiment, it is assumed that the frame rate of the imaging unit is low, so the width of the detection area 401 in the product removal direction (Y-axis direction) is set as shown in FIGS. 8, 17, and 18. However, in the second embodiment, since the frame rate of the imaging unit is high, the width of the detection area 401 in the product removal direction (Y-axis direction) is set to be wider than the detection area 2210 in FIG. Can be set narrowly.

Next, in step S2002, the CPU 201 of the information processing device 102 receives the setting of a detection line corresponding to the entrance/exit of the product shelf based on the acquired captured image. Setting of the detection line will be explained with reference to FIG. 22.

FIG. 22 is a schematic diagram illustrating an image of a product shelf having a plurality of shelves in the height direction and an area (detection area) from which products are taken out in the embodiment of the present invention.

In FIG. 22, 103 is an example of a product shelf, 104 is an example of a product, and 404, which is an imaging range, corresponds to a product outlet. Note that the detection in the horizontal direction (for each lane) of each shelf is the same as in the first embodiment, so it will be omitted in the following description.
In the example of FIG. 22, the product shelf 103 has three shelves.

In the detection area 2210 of FIG. 22, detection lines 2201 to 2203 are set for each of the three shelves. Note that in FIG. 22, the detection lines 2201 to 2203 are shown at different positions to aid understanding, but the detection lines are set at the edge of the detection area 2210 (in the case of FIG. 22, the lower side of the rectangle of the detection area 2210). Each shelf has a different width. The width size is, for example, the width set in FIG. 18. Depending on whether a product has passed through this detection line, it can be determined whether the product has entered or exited from the relevant shelf, preventing false detection of products that have entered or exited from the shelf next to it, or products that have accidentally passed through the detection area (for example, when a customer This can prevent false detections such as when the user holds the camera and crosses the detection area from right to left. Although the embodiment is explained using one inventory monitoring system 105, a plurality of inventory monitoring systems 105 each equipped with an imaging unit 101 are installed, and product detection is performed in the rack of each inventory monitoring system 105. There may also be cases. In this case, it is possible to prevent erroneous detection of products on the side rack. Returning to the explanation of the flowchart in FIG. 20.

Next, when the process moves to step S2003 in FIG. 20, the CPU 201 of the information processing apparatus 102 starts a process for managing product inventory. The processing contents of step S2003 will be explained with reference to FIG. 21.

FIG. 21 is a flowchart showing a processing overview of the second embodiment of product inventory tracking in the information processing apparatus 102 according to the embodiment of the present invention, and S2101 to S2118 in the figure indicate each step. The processing of each step is realized by the CPU 201 of each system loading an application program stored in the external memory 211 of the information processing device 102 onto the RAM 203 and executing it.

The flowchart in FIG. 21 is a flow of processing that is started when the flowchart in FIG. 20 transitions to step S2003.

First, in step S2101 in FIG. 21, the CPU 201 of the information processing apparatus 102 acquires a captured image (capture) from the imaging unit 101. This captured image acquisition may be performed at all times during the processing of the flowchart in FIG. 21 . Note that in the process of step S2101, object detection by the TOF sensor 305 may be performed simultaneously with the acquisition of the captured image.

Next, in step S2102, the CPU 201 of the information processing apparatus 102 determines whether an object such as a product or a hand has been detected within the detection area 2210 within the photographing area 404. If an object is detected, the process moves to step S2103; if no object is detected, the process returns to step S2101, and the process is repeated until an object is detected. Whether or not an object has been detected may be determined by object detection using a TOF sensor, detection based on image difference information, or object detection using a stereo camera.

Next, in step S2103, the CPU 201 of the information processing apparatus 102 specifies the position where the object is detected by the TOF sensor or the like. This location information can be acquired at any time and tracked at all times.

Next, in step S2104, the CPU 201 of the information processing apparatus 102 stores an image around the position where the object is detected. An example of the stored image will be explained with reference to FIG. 23.

FIG. 23 is an example of a series of product images stored in the information processing device 102 according to the embodiment of the present invention.

From 2301 to 2312 in FIG. 23, images stored in chronological order in the order of detected objects are displayed. Specifically, FIG. 23 is a series of images when a product in a plastic bottle is returned to the shelf by a customer. By limiting the range of images to be captured and stored or using them for product identification as in this process, the amount of image storage memory can be reduced compared to the method of identifying objects from the entire image as in the first embodiment. In addition, it is possible to shorten the image processing time when identifying products. . Returning to the explanation of the flowchart in FIG. 21.

Next, in step S2105 of FIG. 21, the CPU 201 of the information processing apparatus 102 acquires height information (depth information) of the object from the TOF sensor or stereo camera.

Next, in step S2106, the CPU 201 of the information processing device 102 determines whether the object has passed through the detection lines 2201 to 2203 from the tracking information acquired in step S2103. Whether the object passes through the detection line is determined based on the height information of the object acquired in step S2105 and the tracking information acquired in step S2103. For example, if the height information of the object is detected as the height of the first shelf (20 cm in the example of FIG. 18), the detection line of the first shelf will be "-" in the size on the screen in the case of FIG. 150-150''. Similarly, when the height information of the object is detected as the height of the third shelf (80 cm in the example of FIG. 18), the detection line of the third shelf is -90 to 90''. If an object passes between these lines, it is determined that it has passed through the detection line. In other words, it means that products or hands have entered and exited the area between the shelves and the front of the shelves. On the other hand, if the height information of the object is detected as the height of the third stage and the horizontal direction of the detection line passes through the position "110", for example, it is determined that the object is being put in or removed from the next shelf, and the corresponding detection It is not considered that the line has been crossed.

If it is determined in step S2106 that the object has passed the detection line, a flag (not shown) indicating that the object has passed the detection line is set, and the process moves to step S2108. At the same time, height information is memorized. Thereafter, while the detection line passing flag is set, the process transitions to Yes in step S2106. On the other hand, if it is determined that the detection line has not been passed, the process transitions to step S2107.

When the process transitions to step S2107, the CPU 201 of the information processing apparatus 102 determines whether the object being tracked has left the detection area 2210 (OUT). If it is determined that you have left the detection area 2210, this means that you have left the detection area without passing through the detection line (in other words, you have not touched the inside of the shelf). The deletion process (step S2118) is executed, and the process returns to the first process. On the other hand, if the object has not come out of the detection area, this means that the object is within the detection area, so the stored image is not deleted and the process returns to the beginning.

When the process transitions to step S2108, that is, when the object passes through the detection line even once, the CPU 201 of the information processing device 102 determines whether the object has exited the detection area in the opposite direction to the detection line. If an object appears in the direction opposite to the detection line, the process moves to step S2111, and if no object appears in the direction opposite to the detection line, the process moves to step S2109. In the example of FIG. 22, whether the object leaves the detection area in the direction opposite to the detection line is determined by whether the object leaves the detection area in the direction of an upward region such as 2211. That is, the process of step S2108 is a process of determining whether a product or a hand has been taken out from the shelf.

When the process transitions to step S2109, the CPU 201 of the information processing apparatus 102 determines whether or not the object has passed through the detection line from the detection area. If an object has come out from the detection line, the process moves to step S2110, and if no object has come out from the detection line, the process moves to step S2117. In the example of FIG. 22, whether the object has come out from the detection line is determined by whether it has passed through the detection lines 2201 to 2203 and left the detection area. That is, the process in step S2109 is a process for determining whether there is a product or a hand on the shelf.

Note that even if the object exits from the detection line, if the object enters from the same detection line and is being tracked, the process transitions to No, and the process transitions to the decision branch of step S2117. The reason for the transition to No is to prevent the product from being counted up, for example, when a customer takes the product from the shelf and returns it to the shelf without taking it out of the detection area.

When the process transitions to step S2110, the CPU 201 of the information processing device 102 sets a count-up flag (not shown) in order to execute processing to increase inventory when the object is a product. Thereafter, the process transitions to step S2112.

On the other hand, when the process transitions to step s2111, the CPU 201 of the information processing apparatus 100 sets a countdown flag (not shown) in order to execute the process of reducing inventory when the object is a product. Thereafter, the process transitions to step S2112.

When the process transitions to step S2112, the CPU 201 of the information processing apparatus 100 reads the plurality of captured images stored in step S2104 and acquires the images.

Next, in step S2113, the CPU 201 of the information processing apparatus 100 performs processing to identify the product through image recognition using AI. Specifically, thousands of images (thousands of images of plastic bottles at various angles when inferring images like the one in Figure 23) are registered as training data for AI, and the data is then used to create AI data such as VGG, MobileNet, etc. It is trained using deep learning algorithm. For example, a plurality of images 2301 to 2312 as shown in FIG. 23 are input to the learned model, and the inferred plastic bottle product is specified. In this way, since one product is inferred from multiple images around an object, the speed of image analysis is faster than the process of identifying a product from one entire image as in the first embodiment, and the image capturing unit You can increase the frame rate. Note that, instead of using AI, the product may be identified by capturing an image of a tag attached to the product, identifying it, and performing matching. In addition, it is also possible to store multiple images and identify the product based on simple matching conditions.In such cases, multiple images around the object can be used to make a judgment, increasing image analysis processing and processing. You can increase your speed.

Next, in step S2114, the CPU 201 of the information processing apparatus 100 determines whether the object specified in step S2113 is a product or not. If it is a product, the process moves to step S2115, and if it is a hand instead of a product, or if the product cannot be specified, the process moves to step S2118.

When the process transitions to step S2118, the CPU 201 of the information processing apparatus 100 deletes the stored image, the height information of the object, the detection line passing flag, the number of increases and decreases in inventory, and the like. The transition to step S2118 occurs when an object leaves the detection area and the detected object is a hand or an unidentifiable product, or when the object leaves the detection area without putting the hand inside the shelf. Since the stored image, object height information, detection line passage flag, and stock increase/decrease number are no longer needed, these data are deleted. After that, the process returns to the beginning of this flowchart. Note that if the user's hand and the unrecognizable product can be identified in step S2113, when an unrecognizable product is placed on the shelf, an alert is issued to notify the store that there is an unrecognizable product. There may also be a configuration to notify the product manager of the product.

On the other hand, when the process transitions to step S2115 as a result of the decision branch in step S2114, the CPU 201 of the information processing device 100 stores the product inferred in step S2114 on the shelf at the height stored in step S2106. Information for increasing or decreasing the inventory by the inventory increase/decrease number in step S2110 or step S2111 is set.

Next, in step S2116, the CPU 201 of the information processing apparatus 100 transmits the shelves, products, and increase/decrease information set in step S2115 to the inventory management server 107, and the process transitions to step S2117.

When the process transitions to step S2117, the process ends when the flowchart of this embodiment is ended or a termination instruction is received from the user (not shown). Return to processing.

As in the above process, object detection and tracking are first performed, and when an object leaves the detection area, the product is identified using the captured image and inventory information is managed. This has the effect of omitting the process of identifying the product from the entire image, increasing the frame rate of the imaging unit, and enabling detailed tracking.

Note that in the second embodiment, it is determined whether the product has been taken out or taken out from the shelf depending on whether the product has passed the detection line. It may be determined that the data has been taken in and taken out. At this time, the height information of the shelf where the product is taken out or taken in is set from the height when the product passes the detection line (step 2106).

Through the above-described processing, the present invention has the effect that it is possible to manage product shelves consisting of a plurality of rows with a small number of imaging devices, and to manage the entry and exit of products regardless of whether or not the stock location is photographed.

Furthermore, in addition to inventory management, data for each shelf height and lateral position can be acquired sequentially, so it also has the effect of being able to acquire information for appropriately allocating shelf allocations.

In addition, since the placement location and entry/exit of products can be specified, it is possible to obtain information that can be used to easily perform planograms suitable for increasing sales.

Furthermore, in order to manage inventory, it is necessary to specify the type of article, but if the type of article is identified every time from the acquired images, processing will be delayed and the frame rate will be reduced. Therefore, it is possible to solve the problem that the detection timing of the article is delayed and it is difficult to track the article.

Furthermore, since the location of the product can be specified based on the {position/height/product image} specified by the image, it is possible to save labor on maintenance of the product placement location by the user, and improve inventory management and product inventory information. It can be done more accurately.

Further, the program according to the present invention is a computer-executable (readable) program for each processing method, and the storage medium according to the present invention stores a computer-executable program for each processing method.

Note that the program in the present invention may be a program for each processing method of each device.

As described above, a recording medium recording a program that implements the functions of the embodiments described above is supplied to a system or device, and the computer (or CPU or MPU) of the system or device reads the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved by reading and executing.

In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium that stores the program constitutes the present invention. Examples of recording media for supplying programs include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, DVD-ROMs, magnetic tapes, non-volatile memory cards, ROMs, EEPROMs, and silicon A disk or the like can be used.

In addition, by executing a program read by a computer, not only the functions of the above-described embodiments are realized, but also the OS, etc. running on the computer performs part of the actual processing based on the instructions of the program. It goes without saying that this also includes a case where all processes are performed and the functions of the above-described embodiments are realized by the processing.

Furthermore, after the program read from the recording medium is written into the memory of the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board It goes without saying that this also includes a case where a CPU or the like provided in a function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

It goes without saying that the present invention can also be applied to a case where the present invention is achieved by supplying a program to a system or device. In this case, by reading a recording medium storing a program for achieving the present invention into the system or device, the system or device can enjoy the effects of the present invention.

Further, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network using a communication program, the system or device can enjoy the effects of the present invention. Note that all configurations that are combinations of the above-described embodiments and their modifications are also included in the present invention.

101 Network camera 102 Information processing device 103 Product shelf 104 Product 105 Inventory monitoring system 106 Network 107 Inventory management server 201 CPU
202 ROM
203 RAM
204 System bus 205 Input controller 206 Video controller 207 Memory controller 208 Communication I/F controller 209 Input section 210 Display section 211 External memory 301 CPU
302 ROM
303 RAM
304 System bus 305 TOF sensor 306 Depth measurement section 307 RGB camera section 308 Image processing section 309 Communication I/F controller

Claims (8)

An information processing device capable of communicating with an imaging unit,
a movement direction specifying means for specifying the direction of movement of the article based on an image taken in the front direction of the shelf by the imaging unit;
an entry/exit specifying means for specifying the exit or entry of the article based on the direction specified by the movement direction specifying means;
An information processing device comprising: a management means for managing the number of articles on the shelf based on the position of the article detected by the article position detection section and the entry/exit specified by the entry/exit specifying means. The position of the article determined by the article position detection unit includes the height of the article entering and exiting, which is specified based on the image obtained from the imaging unit,
2. The management means manages the number of articles corresponding to the shelf position based on the position including the height and the entry/exit specified by the entry/exit specifying means. Information processing device. The shelf is a plurality of rows of shelves,
comprising a shelf specifying means for specifying to which row of the shelves the article is in and out, based on the image obtained from the imaging unit and the position of the article determined by the article position detection unit;
3. The information processing apparatus according to claim 1, wherein the management means manages the number of articles in correspondence with the row determined by the shelf identification means. The shelf is a multi-level shelf,
The shelf identifying means identifies which stage of the shelf the article is in and out of, based on the image obtained from the imaging unit and the position of the article determined by the article position detection unit;
4. The information processing apparatus according to claim 3, wherein the management means manages the number of articles in correspondence with the tiers determined by the shelf identification means. 5. The information processing apparatus according to claim 4, wherein the moving direction of the article specified by the moving direction specifying means is a direction in which the article enters from the shelf or a direction in which the article exits from the shelf. Further comprising a storage means for storing the type of article corresponding to the position of the shelf,
Claims 1 to 5, wherein the number management means manages the number of the types of articles based on the position of the article detected by the article position detection section and the entry/exit specified by the entry/exit specifying means. The information processing device according to any one of the above. An information processing method for an information processing device capable of communicating with an imaging unit, the method comprising:
a moving direction specifying step of specifying the moving direction of the article based on an image taken in the front direction of the shelf by the imaging unit;
an in/out specifying step of specifying whether the article is in or out based on the direction specified in the movement direction specifying step;
An information processing method comprising: a management step of managing the number of articles on the shelf based on the position of the article detected by the article position detection section and the entry/exit specified by the entry/exit specifying step. A program for an information processing device capable of communicating with an imaging unit,
a movement direction specifying means for specifying the direction of movement of the article based on an image taken in the front direction of the shelf by the imaging unit;
an entry/exit specifying means for specifying the exit or entry of the article based on the direction specified by the movement direction specifying means;
A program for functioning as a management means for managing the number of articles on the shelf based on the position of the article detected by the article position detection section and the ingress and egress specified by the ingress and egress specifying means.

Images