JP2015090579A - Behavior analysis system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To analyze/collect customer behavior within shops, and collect an index of the customer behavior.SOLUTION: A behavior analysis system includes: a plurality of imaging devices; and a recognition collection device that acquires data from the plurality of imaging device. The recognition collection device has: a face information database that stores face images photographed by the plurality of imaging devices; and an imaging device information database that is indicative of locations where the plurality of imaging devices are installed, and the recognition collection device has: a face area detection part that detects a face area from the images photographed by the imaging devices; an amount-of-characteristic extraction part that extracts the amount of image characteristic of the detected face area, and registers the amount of characteristic in the face information database; a collation part that determines whether face information registered in the face information database belongs to an identical person on the basis of the amount of image characteristic, and registers information on the identical person on the basis of a determination result in the face information database; and a collection part that collects a behavior index of visitors using data on the time and location where the face information having the information on the identical person provided thereto is photographed.

Description

  The present invention relates to a behavior analysis system, and more particularly to a system suitable for analyzing customer behavior in a store.

  At retail stores, customer behavior in the store is analyzed to derive store improvement measures that lead to increased sales. Previously, surveys were conducted by counting the number of customers entering the store, but recently, a system has been developed to determine the sex and age of customers using face detection technology from images from cameras and other imaging devices. The method of measuring the gender and age counts of customers without using staff is becoming widespread. As another technique, a similar image search technique has been developed that can search for an image with image similarity by digitizing image feature points (image feature amounts) and comparing them.

  As a technique for utilizing such a face recognition result in sales, there is a technique described in Patent Document 1. In Patent Document 1, images are taken at predetermined intervals using a camera function installed in a digital signage device, a vending machine device, and a security monitoring device, face attribute information of the taken images is extracted, and an ID is added in advance. If the comparison result does not match, the ID corresponding to each face information is newly set, and the newly set ID and the detected environment attribute detected this time are stored, If the comparison result is the same, the detected environment attribute is updated with the matched ID and the current detection result and stored. Based on the stored ID, face attribute, and update environment attribute update history information, the background and the person A method for performing identification and controlling a target device is described.

JP 2013-003817 A

  By the way, in order to analyze customer behavior in a store, the number of stores by gender and age is not enough, and there are effective indicators for store improvement measures such as store stay time and repeater rate of visitors and purchasers. There are many. Since the conventional system cannot measure in terms of time, it is difficult to measure these indices. For example, if the system is installed at a cash register, the number of purchasers can be counted, but the purchaser's time spent in the store is not known. Also, it cannot be determined whether the person has purchased in the past.

  Further, in the technique described in Patent Document 1, the device is controlled by switching the display of recommended products in accordance with the recognized face information. However, the photographing apparatus is installed only in one place and installed in a plurality of places. It is not intended to analyze customer trends based on information from the acquired imaging device. For this reason, the behavior of the customer in the store could not be grasped.

  An object of the present invention is to analyze and collect customer behavior in a store by using videos acquired by a plurality of fixed imaging apparatuses, and to aggregate customer behavior indicators such as store stay time.

  A typical example of the invention disclosed in the present application is as follows. In other words, the behavior analysis system includes a plurality of imaging devices and a recognition and aggregation device that acquires data from the plurality of imaging devices, and the recognition and aggregation device stores face images captured by the plurality of imaging devices. A face information database; and a photographing device information database indicating positions where the plurality of photographing devices are installed. The recognition and counting device includes a face region detecting unit that detects a face region from an image photographed by the photographing device; A feature amount extraction unit that extracts an image feature amount of the detected face area and registers the feature amount in a face information database; and face information registered in the face information database based on the image feature amount. A collation unit that determines whether the person is the same person, and registers information that is the same person in the face information database based on the result of the determination, and a face that is provided with the information that is the same person By using the time and location of the data obtained by photographing the broadcast, with a, and aggregation unit to aggregate the behavior index of coming to the store's.

  According to a typical embodiment of the present invention, it is possible to add up an index of customer behavior in a store.

It is a system configuration figure showing an example of an embodiment of the present invention. It is a figure explaining the structure of the data stored in the face information database 10 of this embodiment. It is a flowchart of the same person collation part process of this embodiment. It is a figure explaining the structure of the imaging device information stored in the imaging device information database 11 of this embodiment. It is a flowchart of the total process of this embodiment. It is a figure explaining the output content of the output list of this embodiment. It is a figure explaining the output content of the output list of this embodiment. It is a figure explaining the output content of the output list of this embodiment. It is a figure explaining the output content of the output list of this embodiment.

  Hereinafter, an embodiment of a customer trend time counting system to which the present invention is applied will be described.

  FIG. 1 is a system configuration diagram showing an example of an embodiment of the present invention.

  The customer trend time tabulation system of this embodiment includes a plurality of photographing devices 1a, 1b,. The imaging devices 1a, 1b,..., 1n may be those that shoot moving images or those that repeatedly shoot still images. The photographing devices 1a, 1b,..., 1n are connected to the network 2 by wireless lines (or wired lines), and the recognition / aggregation apparatus 3 is connected to the network 2. Therefore, the photographing devices 1a, 1b,..., 1n and the recognition / aggregation device 3 can transfer data through the network 2.

  The recognition tabulation device 3 is connected to the terminal device 20 via the network 2. The terminal device 20 sends an instruction to the recognition / aggregation device 3 or displays a calculation result by the recognition / aggregation device 3.

  The recognition and totalization device 3 is connected to the POS system via the network 2. The POS system includes a POS server 30, a POS terminal 31, and a card reader 32. The POS server 30 counts sales data input to the POS terminal 31. The POS terminal 31 is installed at an accounting place and sales data is input. The card reader 32 is connected to the POS terminal 31 and reads a storage medium (for example, a membership card, a point card). The recognition and aggregation device 3 can reliably identify the purchaser based on the data acquired from the POS system.

  In addition, at least one of the photographing devices 1a, 1b,..., 1n may be provided at a checkout location (preferably for each lane of the cash register) to capture an image of a customer who has passed the cash register.

  The recognition and aggregation device 3 is a computer having a processor that executes a program, a memory, a storage device, and a communication interface.

  The memory is a high-speed and volatile storage element such as a DRAM (Dynamic Random Access Memory), and temporarily stores a program stored in the storage device and data used when the program is executed. A storage device, for example, a large-capacity nonvolatile storage device such as a magnetic storage device or a flash memory, and stores a program executed by the processor and data used when the program is executed. That is, the program is read from the storage device, loaded into the memory, and executed by the processor.

  The communication interface is a network interface device that controls communication with other devices connected via the network 2 in accordance with a predetermined protocol.

  A program executed by the processor is provided to each server via a removable medium (CD-ROM, flash memory, etc.) or a network, and is stored in a nonvolatile storage device which is a non-temporary storage medium. Therefore, the computer system may have an interface for reading data from the removable medium.

  The recognition and aggregation device 3 is a computer system configured on a single computer or a plurality of logically or physically configured computers, and operates in a plurality of threads on the same computer. Alternatively, it may operate on a virtual machine constructed on a plurality of physical computer resources.

  The recognition and aggregation device 3 stores the face information database 10, the imaging device information database 11, and the aggregation result storage unit 12 in a storage device.

  The face information database 10 stores face acquisition information such as face image feature data, face area coordinates, and shooting date / time. Details of the face information database 10 will be described later with reference to FIG. The photographing device information database 11 stores information of each photographing device such as photographing locations by connected photographing devices 1a, 1b, ..., 1n. Details of the photographing apparatus information database 11 will be described later with reference to FIG. The tabulation result storage unit 12 stores customer behavior indexes tabulated by the tabulation unit 9.

  The recognition / aggregation apparatus 3 includes an image acquisition unit 4, a face area detection unit 5, a face image feature amount extraction unit 6, a face information database registration unit 7, an identical person verification unit 8, and a totalization unit 9. Each of these units is implemented by a processor executing a predetermined program.

  The image acquisition unit 4 acquires images from the connected imaging devices 1a, 1b,. The face area detection unit 5 detects a face area from the acquired image. The face image feature amount extraction unit 6 quantifies the image feature amount of the face area portion of the acquired image. The face information database registration unit 7 registers the face image feature amount and face acquisition information in the face information database 10. The same person collating unit 8 collates data of the same person from the face information database 10 in which the face information is accumulated, and gives the same person information to the face information database 10. The totaling unit 9 acquires the time when the same person was photographed and the photographing device ID photographed from the face information database 10 to which the same person information was assigned, and obtains the installation location of the photographing device from the photographing device information database 11. , Aggregate customer behavior indicators from the acquired information.

  FIG. 2 is a diagram illustrating the configuration of data stored in the face information database 10 of the present embodiment.

  The face information database 10 is a database that stores face image feature value data and face acquisition information. As shown in FIG. 2A, the face image feature value 21, the shooting date and time 22, the face area coordinates 23, the person ID 24, and the shooting information. Data of the device ID 25 is registered.

  As the face image feature value 21, data digitized by the face image feature value extraction unit 6 is registered. The shooting date 22 registers the shooting date of the shot image. As the face area coordinates 23, as shown in FIG. 2B, the coordinates of the face area detected by the face area detecting unit 5 are registered using coordinates with the upper left of the photographed image 26 as the origin.

  In addition, the value of the person ID 24 is not registered at the time of registration in the face information database 10, but is registered by the same person verification unit 8. Further, as the photographing apparatus ID 25, an identifier of the photographing apparatus that has photographed the image where the face is detected is stored, and the same one as the photographing apparatus ID 41 registered in the photographing apparatus information database 11 is used. These pieces of information are collected to form one record and represent one piece of face information.

  FIG. 3 is a flowchart of the same person verification unit process of the present embodiment. The same person collation process shown in FIG.

  First, the face information database registration unit 7 sets a new person ID to 0 (step 301). When the image acquisition unit 4 acquires a new captured image from the imaging device 1 (step 302), the face area detection unit 5 detects the face area. At this time, if the face area cannot be detected (N in step 303), the process returns to step 302 to acquire the next photographed image. When the face area can be detected (Y in Step 303), the face image feature amount extraction unit 6 extracts the digitized face image feature amount from the face image (Step 304), and the face information database registration unit 7 The image feature quantity 21, the shooting date / time 22, the face area coordinates 23, and the shooting apparatus ID 25 are registered in the face information database 10 (step 305).

  Thereafter, the same person verification unit 8 searches the face information database 10 for a face information record having a face image feature amount similar to the face image feature amount registered in step 305 (step 306). When a face information record having a similar face image feature amount is not found (N in Step 307), the person ID 24 of the face information record registered this time is updated with a new person ID (Step 310), and the new person ID is incremented by one. (Step 311), returning to Step 302, the next captured image is acquired. On the other hand, when a face information record having a similar face image feature amount is found (Y in step 307), it is determined that the face information record registered this time and the face information record of the search result are the same person (step 308). ) The person ID 24 of the face information record registered this time is updated using the person ID of the face information record of the search result (step 309). The above processing is repeated every time a captured image is acquired.

  FIG. 4 is a diagram illustrating the configuration of the imaging device information stored in the imaging device information database 11 of the present embodiment.

  The imaging device information database 11 includes an imaging device ID 41 and an imaging location 42, and the imaging location 42 is registered for each imaging device ID 41. For example, the imaging apparatus information shown in FIG. 4 indicates that an imaging apparatus having an imaging apparatus ID 41 of 0 is installed so as to reflect the entrance.

  FIG. 5 is a flowchart of the aggregation process of the present embodiment. The counting process shown in FIG. 5 is executed by the counting unit 9 of the recognition and counting device 3.

  First, the counting unit 9 sets the search person ID to 0 (step 501). Then, face information whose person ID matches the search person ID is searched from the face information database 10 (step 502). If face information whose search person ID and person ID match is not found (N in step 503), the search person ID is incremented by 1 (step 508), the process returns to step 502, and the face information database 10 is stored with the next search person ID. Search for.

  On the other hand, when face information whose search person ID and person ID match is found (Y in step 503), the shooting date 22 of the face information record is obtained from the face information database 10, and the shooting date of the found face information is set. Specify (step 504). Further, the photographing device ID 25 of the face information record is acquired from the face information database 10, the photographing device 42 is obtained by referring to the photographing device information database 11 with the obtained photographing device ID, and the photographing location of the found face information is specified. (Step 505).

  Then, the moving time between the shooting locations of the person with the retrieved person ID is calculated from the acquired shooting date and shooting location (step 506). For example, the store stay time can be calculated by subtracting the time taken at the entrance from the time taken at the exit. Specifically, the set of the acquired shooting date and shooting location (2013/07/25 10:00:00, entrance), (2013/07/25 10:28:00, cash register), (2013/07 / 25 10:30:00, Exit), this person has purchased a product, and the store stay time from entering the store to the store is 30 minutes, and the store stay time is known. For example, if the pair of the acquired shooting date and time and the shooting location is (2013/07/25 10:00:00, entrance) and (2013/07/25 10:30:00, exit), this person Has not purchased anything, and the store stay time from entering the store to the store is 30 minutes, so the store stay time is known.

  Then, the calculated information is output as a list and stored in the tabulation result storage unit 12 (step 507), the search person ID is incremented by 1 (step 508), and the process returns to step 502 to return to the face information database 10 with the next search person ID. Search for. By repeating the above processing for all person IDs, retail store customer trend time information of all customers can be acquired from the face information stored in the face information database 10.

  FIG. 6 is a diagram for explaining the output contents of the output list of this embodiment.

  The store stay time is calculated by subtracting the time taken at the entrance from the time taken at the exit. In the data illustrated in FIG. 6, four people visited the store, two people purchased the product, the average store stay time of the visitor was 17.25 minutes, and the average store visit time of the purchaser was 12 minutes. is there.

  In addition, when photographing at the exit is performed a plurality of times with respect to photographing at one entrance, the store stay time may be calculated using the latest exit photographing time.

  Further, if a visitor who has made a checkout at the cashier shooting time uses a member card or a point card, the visitor can be reliably identified by the identifier (membership number or the like) of the read card.

  In the above-described embodiment, an example in which the stay time in the customer store is acquired has been described. However, various customer behavior indicators can be measured by changing the installation location of the imaging device.

  Other examples are given below.

  As shown in FIG. 7, the shooting date and time within a predetermined period may be listed in the output list for each search person ID with the shooting location as the entrance. A person filmed multiple times is a repeater customer. The number of times the person has been photographed is aggregated to calculate a visitor repeater rate indicating whether the store visitor is a repeater and a purchaser repeater rate indicating whether the purchaser is a repeater. In the data illustrated in FIG. 7, four people come to the store, one person has purchased the product, the visitor repeater rate (number of repeat visitors / total number of visitors) is 0.5, and the purchase repeater rate (repeat visitor) (Number / total number of purchasers) is 1.0.

  In addition, the place which totals the imaging | photography frequency for every search person ID may be arbitrary places. By such aggregation, the repeater rate at a specific place can be calculated. Further, by changing the period for counting the number of times of photographing, it is possible to calculate the repeater rate of the visitor during a specific period (for example, during a sale).

  As shown in FIG. 8, the shooting location may be an entrance, a cash register, and a priority product shelf, and three shooting dates and times may be listed for each search person ID. In this case, a photographing device is installed in a specific product shelf, and a store visitor facing the product shelf is photographed. It is presumed that the person photographed in the priority product shelf was interested in the priority product. For this reason, an important product attracting rate representing a visitor who pays attention to the important product among the total number of visitors or purchasers is calculated. This gives an indication of whether the customer is paying attention to the priority product. In the data illustrated in FIG. 8, four people visit the store, two people purchase the product, the priority product attracting rate among the visitors is 0.5, and the important product attracting rate among the purchasers is 0. 0.

  Further, by combining the cash register shooting time and POS data with the priority product shelf shooting time, it is possible to know whether or not the customer has purchased the product. In other words, if the priority product is included in the accounting at the cash register photographing time, it can be understood that the person who stopped at the priority product shelf purchased the priority product. In this case, a result of comparing the result captured on the priority product shelf with the purchase data of the priority product is output to the output list so that a visitor who stops at the priority product shelf purchases the priority product. .

  As shown in FIG. 9, the shooting location may be an entrance, a cash register, and a priority passage, and three shooting dates and times may be listed for each search person ID. In this case, a photographing device is installed so as to photograph from both sides of the important passage (or the entrance and exit of the important passage), and the store visitor who has passed through the important passage is photographed. The number of persons photographed in the priority path, that is, the persons who have passed through the priority path, is totalized, and among the total number of visitors or purchasers, the priority path customer passing rate representing the shoppers who have passed through the priority path is calculated. This provides an indication of whether the customer is passing through the priority passage. In the data illustrated in FIG. 9, four people visit the store, two people purchase the product, the priority passage customer passage rate of the visitor is 0.75, and the priority passage customer passage rate of the purchaser is 1.0.

  In addition, a photographing device is installed so as to photograph from both sides of the priority passage (or the entrance and exit of the priority passage), and a visitor who has passed through the priority passage is photographed. The direction of passing through the priority passage can be known by the order of the time taken at the two points of the priority passage.

  As described above, in the embodiment of the present invention, the staying time of the store visitor / purchaser can be easily measured. In addition, it is possible to easily calculate a visitor repeater rate indicating whether a visitor is a repeater, and a purchaser repeater rate indicating whether a visitor is a repeater. In addition, it is possible to easily calculate the priority product attracting rate indicating whether the store visitor or the purchaser pays attention to the priority product. Furthermore, it is possible to easily calculate an important passage customer passing rate that indicates whether the store visitor or the purchaser passes the important passage.

  Further, by using the cash register shooting time and the POS data in combination, it is possible to know whether the customer has purchased the product. That is, it is possible to analyze the customer behavior in the store and the purchase behavior in association with each other.

  Furthermore, a store visitor can be reliably specified by using together the data acquired from the member card or the point card.

DESCRIPTION OF SYMBOLS 1a, 1b, 1n Image pick-up device 2 Network 3 Recognition totalization device 4 Image acquisition part 5 Face area detection part 6 Face image feature-value extraction part 7 Face information database registration part 8 Single person collation part 9 Total part 10 Face information database 11 Image pick-up apparatus Information database

Claims (5)

A behavior analysis system comprising a plurality of imaging devices and a recognition and counting device that acquires data from the plurality of imaging devices,
The recognition and counting device includes a face information database that stores face images taken by the plurality of photographing devices, and a photographing device information database that indicates locations where the plurality of photographing devices photograph,
The recognition and counting device
A face area detection unit that detects a face area from an image captured by the photographing apparatus;
A feature amount extraction unit that extracts an image feature amount of the detected face area and registers the image feature amount in the face information database;
Based on the image feature amount, it is determined whether the face information registered in the face information database belongs to the same person, and the information on the same person is registered in the face information database based on the determination result. A collation unit;
A behavior analysis system, comprising: a totaling unit that counts a behavior indicator of a store visitor using data of time and photographing location when photographing face information to which the information of the same person is given. The behavior analysis system according to claim 1,
The plurality of imaging devices includes a first camera that images an entrance of a store, a second camera that images an exit of the store, and a third camera that images an accounting location of the store,
The recognition and counting device
The stay time is calculated by the difference between the time when the face image was taken with the second camera and the time when the face image was taken with the first camera,
The behavior analysis system characterized in that the calculated staying time and the time when the face image was taken by the third camera are output. The behavior analysis system according to claim 1,
The plurality of imaging devices include a first camera that images an entrance of a store, and a third camera that images an accounting location of the store,
The recognition and counting device
Count the number of times the same camera's face image was taken with the first camera within a predetermined period,
Count the number of times the same camera's face image was taken with the third camera within the predetermined period,
A behavior analysis system that outputs the counted number of times of photographing. The behavior analysis system according to claim 1,
The plurality of imaging devices include a third camera that images a store accounting location, and a fourth camera that images a specific location in the store,
The behavior analysis system is connected to a POS system including a POS terminal installed at the accounting place,
The recognition and counting device
Determine whether a face image of a visitor was taken with the fourth camera,
Obtain the time when a face image of a visitor was taken by the third camera,
From the accounting data input to the POS terminal at the acquired time, acquire purchase data of products sold at the specific location,
A behavior analysis system that outputs a result of comparing a determination result of a face image photographed by the fourth camera and the acquired purchase data. The behavior analysis system according to claim 1,
The plurality of photographing devices include a third camera that photographs a store accounting place,
It is connected to a POS system in which a POS terminal is installed at the accounting place,
The POS terminal has a reading device for reading identification information for identifying a visitor,
The recognition and counting device is configured to detect a face image taken by the third camera based on the identification information read by the reading device at or near the time when the face image of a visitor was taken by the third camera. Behavior analysis system characterized by identifying people.

Images