JP7081310B2 - Behavioral analytics device, behavioral analytics system, behavioral analytics method, program and recording medium - Google Patents

Description

The present invention relates to a behavior analysis device, a behavior analysis system, a behavior analysis method, a program and a recording medium for analyzing the behavior of a customer in a store.

Traditionally, the purchasing behavior of customers in stores has been analyzed for the purpose of marketing and sales promotion. For example, in the system disclosed in Patent Document 1, a person is detected from image information, and for the above person, a region of interest (for example, a head or a shoulder) set on a portion of the upper body excluding the hands and arms is detected. .. Then, based on the displacement state of the attention portion, it is determined whether or not the person has performed an article acquisition action of picking up the article arranged in the article arrangement area. When performing an article acquisition action, the posture of the person changes according to the action of reaching for the article, and in particular, the upper body moves. Therefore, by paying attention to the displacement state of the attention portion, it is possible to determine the presence or absence of the article acquisition behavior even in the situation where the movement of the person's hand or arm cannot be detected, thereby analyzing the article acquisition behavior. I am trying to do it accurately.

Further, for example, in the system disclosed in Patent Document 2, when acquiring analysis information regarding a customer's product acquisition behavior, it is determined whether the main body of the action of reaching for the display area is a clerk or a customer, and the clerk's By eliminating the operation and generating the analysis information, the analysis information regarding the customer's product acquisition behavior is accurately acquired.

JP-A-2015-11649 (see claims 1, 2, paragraphs [0008], [0013], FIG. 1, etc.) Japanese Unexamined Patent Publication No. 2016-224800 (see claims 1, 2, paragraphs [0005], [0006], [0010], FIG. 1 and the like).

However, in the system of Patent Document 1, since the presence or absence of the article acquisition behavior is determined based on the displacement state of the attention portion, the attention portion is displaced even when the picked-up article is returned to the original arrangement area, for example. However, it will be judged as an article acquisition behavior. Further, even if the article is not picked up by simply reaching out to the article and coming into contact with the article, if the attention portion is displaced, it is determined as the article acquisition behavior. That is, in the system of Patent Document 1, it is not possible to distinguish whether the article is actually picked up, returned, or just in contact with the article, and the behavior related to the purchase of the article (purchasing behavior). Cannot be determined accurately.

Further, in the system of Patent Document 2, it is only determined whether the main body of the action of reaching for the display area is a store clerk or a customer, and whether the customer actually picked up the product, returned the product, or the product. It is not possible to determine whether the product is just touched, and it is also not possible to accurately determine the customer's purchasing behavior.

The present invention has been made to solve the above problems, and an object thereof is to determine whether the customer has picked up the product, returned the product to its original position, or just touched the product. It is an object of the present invention to provide a behavior analysis device, a behavior analysis system, a behavior analysis method, a program, and a recording medium capable of accurately determining a customer's purchasing behavior.

The behavior analysis device according to one aspect of the present invention is a behavior analysis device that analyzes the behavior of a customer in the store, and estimates the attitude of the customer in the store based on an image taken inside the store. The attitude estimation unit that acquires the attitude information, the product variation information extraction unit that analyzes the display state of the products in the store and extracts the product variation information indicating the variation of the display state from the image, and the attitude. It is provided with a purchasing behavior determination unit that determines the purchasing behavior of the customer based on the information and the product fluctuation information.

The behavior analysis system according to another aspect of the present invention includes the above-mentioned behavior analysis device and an image pickup unit that photographs the inside of the store and acquires the image.

The behavior analysis method according to still another aspect of the present invention is a behavior analysis method for analyzing the behavior of a customer in the store, and the attitude of the customer in the store is based on an image taken inside the store. Product fluctuation information extraction that analyzes the display state of the products in the store based on the posture estimation process that estimates and acquires the posture information and extracts the fluctuation of the display state of the products as the product fluctuation information. The process includes a purchasing behavior determination process for determining the purchasing behavior of the customer based on the attitude information and the product fluctuation information.

The program according to still another aspect of the present invention is a program that causes a computer to execute the above-mentioned behavior analysis method.

The recording medium according to still another aspect of the present invention is a computer-readable recording medium on which the above program is recorded.

Whether the customer actually took the product in order to judge the customer's purchasing behavior based on the customer's attitude information in the store and the product fluctuation information showing the actual fluctuation of the display state of the product in the store. It is possible to determine whether the product has been returned to its original position or just touched the product, thereby accurately determining the customer's purchasing behavior.

It is a block diagram which shows typically the whole structure of the behavior analysis system of one Embodiment of this invention. It is explanatory drawing which shows typically the state in the store to which the said behavior analysis system is applied. It is explanatory drawing which shows an example of the processing flow in the said behavior analysis system. It is explanatory drawing which shows an example of the image of an arbitrary frame acquired by the image pickup part installed in the ceiling of the said store. It is explanatory drawing which shows typically the posture of the customer included in the said image. It is explanatory drawing which shows typically the image of the frame before a customer stays in front of a shelf. It is explanatory drawing which shows typically the image of the frame after a customer stays in front of a shelf. It is a block diagram schematically showing the structure of the behavior analysis system of another embodiment of this invention. It is explanatory drawing which shows typically the product map which shows the correspondence relation between the product information of each product, and the display place. It is explanatory drawing which shows an example of the processing flow in the said behavior analysis system. It is explanatory drawing which shows an example of the product list schematically.

<Embodiment 1>
An embodiment of the present invention will be described below with reference to the drawings.

[Structure of behavior analysis system]
FIG. 1 is a block diagram schematically showing the overall configuration of the behavior analysis system 50 of the present embodiment, and FIG. 2 is an explanatory diagram schematically showing the inside of the store 100 to which the behavior analysis system 50 is applied. Is. The behavior analysis system 50 is a system that analyzes the behavior of the customer P in the store 100, and has a behavior analysis device 1 and an image pickup unit 2.

The image pickup unit 2 is composed of a camera that photographs the inside of the store 100 and acquires an image. The above image is, for example, a moving image, but may be a still image. As shown in FIG. 2, the image pickup unit 2 is installed on the ceiling 101 in the store 100, but any place can photograph the customer P and the product 201 displayed on the product shelf 102. It may be installed in various places (for example, on the product shelf 102 facing the product shelf 102 to be photographed). Further, in the present embodiment, the number of the image pickup unit 2 installed is one, and one image pickup unit 2 captures the customer P and the product 201 in the store 100, but the image pickup unit 2 is the store 100. There may be more than one installed inside. Then, the image of the customer P and the image of the product 201 may be performed by separate image pickup units 2.

The image pickup unit 2 is communicably connected to the behavior analysis device 1 via a communication line such as a LAN (Local Area Network), but may be configured to be capable of wireless communication with the behavior analysis device 1. In any case, the image data acquired by the image pickup unit 2 is sent to the behavior analysis device 1 via the communication line.

The behavior analysis device 1 is a terminal device that analyzes the behavior of the customer P in the store 100, and is composed of, for example, a personal computer (PC). The behavior analysis device 1 includes a posture estimation unit 11, a product change information extraction unit 12, a purchase behavior determination unit 13, a storage unit 14, an input unit 15, a display unit 16, a communication unit 17, and a reading unit. It has 18 and a control unit 19. The control unit 19 is composed of a central processing unit (CPU) that controls the operation of each unit of the behavior analysis device 1, and operates according to an operation program stored in the program storage unit 14a described later.

The posture estimation unit 11 estimates the posture of the customer P in the store 100 and acquires the posture information based on the image in the store 100 acquired by the image pickup unit 2 and input to the behavior analysis device 1. The posture estimation unit 11 has a built-in time measuring unit (not shown) such as a timer, and acquires the posture information for each customer P in the store 100 in chronological order (at different timings in time). The posture estimation unit 11 is composed of, for example, a GPU (Graphics Processing Unit). The GPU is an arithmetic unit (processor) specialized for real-time image processing. By configuring the posture estimation unit 11 with a GPU, it is possible to construct a neural network capable of machine learning such as CNN (Convolutional Neural Network) and estimate the posture with respect to an image input. The posture estimation unit 11 may be configured by the same or separate CPU as the control unit 19.

The product variation information extraction unit 12 analyzes the display state of the product 201 in the store 100 from the image in the store 100 acquired by the image pickup unit 2 and input to the behavior analysis device 1, and changes in the display state. The product fluctuation information indicating is extracted. The product variation information extraction unit 12 is configured by a CPU separate from the control unit 19, but may be configured by the same CPU.

The purchasing behavior determination unit 13 determines the purchasing behavior of the customer P based on the posture information acquired by the attitude estimation unit 11 and the product variation information extracted by the product variation information extraction unit 12. The purchasing behavior determination unit 13 is configured by a CPU separate from the control unit 19, but may be configured by the same CPU. As the above-mentioned purchasing behavior, here, the customer P simply takes out the product 201 from the display place (for example, the product shelf 102), returns the taken-out product 201 to the display place, and reaches out and contacts the product 201. Behavior (contact behavior) can be considered.

The details of each process by the posture estimation unit 11, the product change information extraction unit 12, and the purchase behavior determination unit 13 will be described later in the operation description.

The storage unit 14 is a memory for storing various types of information, and is composed of, for example, a hard disk, but is a recording medium such as a RAM (Random Access Memory), a ROM (Read Only Memory), an optical disk, a magneto-optical disk, or a non-volatile memory. It may be configured by appropriately selecting from. The storage unit 14 has a program storage unit 14a, a posture information storage unit 14b, an image storage unit 14c, and a determination result storage unit 14d.

The program storage unit 14a is a memory for storing an operation program for operating each unit of the behavior analysis device 1. The posture information storage unit 14b stores the posture information estimated and acquired by the posture estimation unit 11. The image storage unit 14c stores image information (image data) of each frame that is different in time and is acquired by the image pickup unit 2 and input to the behavior analysis device 1. From the viewpoint of personal information protection, it is not appropriate to permanently store the image acquired by the imaging unit 2, so that the image information stored in the image storage unit 14c is stored after a predetermined period (for example, after several days). It will be erased after a few months) or overwritten with new image information. The determination result storage unit 14d stores the result (purchasing behavior information) determined by the purchasing behavior determination unit 13.

The input unit 15 is composed of, for example, a keyboard, a mouse, a touch pad, a touch panel, and the like, and receives various instruction inputs by the administrator of the behavior analysis system 50. The display unit 16 is a device that displays various types of information, and is composed of, for example, a liquid crystal display device. The communication unit 17 is an interface for communicating with an external terminal (for example, a mobile terminal such as a smartphone or another PC), and includes an input / output port, an antenna, a transmission / reception circuit, a modulation circuit, a demodulation circuit, and the like. To. The reading unit 18 is a device that reads information recorded on the recording medium R, and is composed of, for example, a disk drive. When the recording medium R is a portable non-volatile memory, the reading unit 18 also includes a connection port into which the connection unit of the non-volatile memory is pointed.

[Behavioral analysis system operation]
Next, the operation of the behavior analysis system 50 of the present embodiment will be described. FIG. 3 is an explanatory diagram showing an example of a processing flow in the behavior analysis system 50 of the present embodiment. The behavior analysis method realized by the behavior analysis system 50 of the present embodiment includes an image acquisition step (S1), a posture estimation step (S2), a product variation information extraction step (S3), and a purchasing behavior determination step (S4). Hereinafter, each step will be described.

(S1; image acquisition process)
In S1, the inside of the store 100 (see FIG. 2) is photographed by the image pickup unit 2, and images for each frame different in time are acquired. FIG. 4 shows an example of an image M of an arbitrary frame acquired by the image pickup unit 2. In the image M, a state (behavior) in which the customer P reaches for the product 201 displayed on the product shelf 102 in the store 100 is shown. The acquired image data of each frame is input to the behavior analysis device 1 and sequentially stored in the image storage unit 14c.

(S2; posture estimation process)
In S2, the posture estimation unit 11 estimates the posture of the customer P in the store 100 and acquires the posture information based on the image acquired in S1, and obtains the posture information for each customer P in the store 100. Get in chronological order. The posture information is based on, for example, the posture when the customer P reaches for the product 201 on the product shelf 102, the posture when the customer P touches the product 201, and the hand extended by the customer P. Information such as the posture when the customer P is returned, the posture when the customer P is crouching, and the posture when the customer P is walking is included. The acquired posture information is stored in the posture information storage unit 14b.

FIG. 5 shows the posture 301 (see thick line) of the customer P included in the image M shown in FIG. The posture information indicating the posture 301 of the customer P includes information on the position of each joint of the body of the customer P (for example, position coordinates in a two-dimensional or three-dimensional space). In the figure, in order to make the posture 301 of the customer P easy to understand, the joint positions are connected by a line, and such a line is also included in the above-mentioned posture information.

The acquisition of the posture information (acquisition of each joint position of the customer P from the captured image) can be performed by using, for example, known image processing software. Further, since the operation range (for example, the movement range) of the customer P within the predetermined time is limited, the movement amount of each joint position (or the line connecting the joint positions) within the predetermined time is within the predetermined range. By determining whether or not there is, it is possible to determine whether the posture information acquired at different timings in time is information about the same customer P or information about different customers P. As a result, the posture information can be acquired separately for each customer P.

(S3; Product fluctuation information extraction process)
In S3, the product variation information extraction unit 13 analyzes the display state of the product 201 in the store 100 from the image of each frame acquired in S1 and extracts the product variation information.

FIG. 6 is an image taken by the image pickup unit 2, and schematically shows the image M1 of the frame before the customer P stays in front of the product shelf 102 (or the product 201 displayed on the product shelf 102). Shows. Further, FIG. 7 is an image taken by the image pickup unit 2, and schematically shows an image M2 of the frame after the customer P stays in front of the product shelf 102. In addition, in FIGS. 6 and 7, for convenience, each product 201 is designated by a reference numeral “1” to “12” to distinguish each product (assuming that the products in each row are of the same type). .. From these images M1 and M2, after the customer P stays in front of the product shelf 102, the product 201 having the codes “3” and “10” is reduced from the product shelf 102 one by one. You can see that. Therefore, the product variation information extraction unit 12 compares the two images M1 and M2 and analyzes the display state of the product 201, so that the product variation information (reference numerals “3” and “10” can be obtained from these images M1 and M2. "Information that the number of products 201 has decreased by one) can be extracted.

Further, the product fluctuation information extraction unit 12 determines whether or not the customer P is staying in front of the product shelf 102 (or the product 201 displayed on the product shelf 102) based on the attitude information acquired by the attitude estimation unit 11. Can be determined. For example, if the same posture 301 of the customer P continues for several frames in front of the product shelf 102, or if the posture 301 changes without changing the position with respect to the product shelf 102, the customer P is in front of the product shelf 102. It can be determined that you are staying at. Therefore, the product fluctuation information extraction unit 12 determines whether or not the customer P has stayed in front of the product 201 based on the above-mentioned attitude information, and if it is determined that the customer P has stayed, the image storage is as described above. Product fluctuation information can be extracted by comparing the image M1 of the frame before the stay stored in the unit 14c with the image M2 of the frame after the stay (in the future) and analyzing the display state of the product 201. can.

(S4; Purchasing behavior determination process)
In S4, the purchasing behavior determination unit 13 determines the purchasing behavior of the customer P based on the attitude information of the customer P acquired in S2 and the product fluctuation information acquired in S3. More specifically, the purchasing behavior determination unit 13 first reaches out to the customer P's access behavior to the product 201 (the customer P reaches for the displayed product 201) based on the change in the posture information acquired in S2. Behavior) is detected. Then, the purchasing behavior determination unit 13 determines the purchasing behavior of the customer P based on the above access behavior and the product fluctuation information extracted in S3.

For example, if the access action is once and the number of products 201 on the product shelf 102 is reduced by one before and after one access action, the customer P has one access action from the product shelf 102 to the product 201. It can be determined that one is taken. The “before and after the access behavior” refers to the relationship between the frame before the customer P stays in front of the product shelf 102 on which the access behavior is performed and the frame after the stay. Further, if there are two access actions in front of the same product shelf 102 and the number of products 201 on the product shelf 102 decreases by two before and after the two access actions, the customer P has the product shelf in each access action. It can be determined that the product 201 is taken one by one from 102. That is, in these cases, it can be determined that the purchase behavior of the customer P is the behavior of taking out the product 201 from the display place.

Further, if there is one access action and the product 201 on the product shelf 102 does not increase or decrease before or after the one access action, the customer P only contacts the product 201 in one access action. It can be determined that the product 201 was not taken. That is, in this case, it can be determined that the customer P's purchasing behavior is such that the product 201 is in contact with the product 201 (contact behavior).

Further, if there are two access actions in front of the same product shelf 102 and there is no increase / decrease in the product 201 on the product shelf 102 before and after the two access actions, the first access action is performed from the product shelf 102. It can be determined that there is a high possibility that the product 201 taken is returned to the product shelf 102 in the second access action. That is, in this case, it can be determined that the purchase behavior of the customer P is the behavior of returning the product 201 to the display place. It cannot be denied that the customer P may not have taken the product 201 only by contacting the product 201 in each of the two access actions, but accessing the product twice means that he / she is interested in the product 201. Therefore, it is highly unlikely that the product 201 will not be picked up just by touching it, especially in the second access. Therefore, as described above, it is appropriate to determine that the purchase behavior of the customer P is the behavior of returning the product 201 to the display place.

The determined purchasing behavior information is stored in the determination result storage unit 14d. As a result, for example, the system administrator can access the behavior analysis device 1 and refer to the purchase behavior information stored in the determination result storage unit 14d to grasp the purchase behavior of the customer P for future marketing. It will be possible to make use of it.

〔effect〕
As described above, in the present embodiment, the purchasing behavior determination unit 13 determines the purchasing behavior of the customer P based on the posture information acquired in S2 and the product fluctuation information extracted in S3 (S4). Just by judging the attitude of customer P (for example, just by judging that customer P reached out to product 201), whether customer P took product 201 from the display location or returned product 201 to the original display location. It is not possible to determine whether the product 201 has just been touched. However, by further considering the information indicating the actual fluctuation of the product 201 (product fluctuation information) in addition to the customer P's attitude information, the movement (behavior) according to the customer P's attitude can be caused by the customer P from the display location to the product. It is possible to determine whether the action is the action of taking the 201, the action of returning the product 201 once taken out to the original display place, or the action of just touching the product 201. As a result, the purchasing behavior of the customer P can be accurately determined.

Further, the product fluctuation information extraction unit 12 determines whether or not the customer P has stayed in front of the product 201 based on the attitude information, and if it is determined that the customer P has stayed, the image M1 of the frame before staying. And the image M2 of the frame after staying are compared with each other, and the product fluctuation information is extracted by analyzing the display state of the product 201 (S3). By comparing each frame image before and after the customer P stays in front of the product 201, it is possible to clearly grasp the change in the display state of the product 201, that is, the increase / decrease in the product 201. As a result, highly reliable product fluctuation information can be obtained.

Further, for example, if the customer P is staying in front of the product 201, it cannot be determined whether or not the customer P has taken the product 201 because the customer P's hand is hidden behind the customer's body in the frame image during the stay. In some cases. However, as described above, by comparing each frame image before and after the customer P stays in front of the product 201, the display state of the product 201 regardless of whether or not the customer P's hand is hidden by the customer's body. You can grasp the change of. Therefore, in this respect as well, highly reliable product fluctuation information can be obtained.

Further, the purchasing behavior determination unit 13 detects the access behavior of the customer P to the product 201 based on the change in the posture information (S4). When purchasing a product 201, there is always an action of accessing the displayed product 201 (reaching out) and taking out the product 201 from the display location (adding it to the cart). Therefore, by detecting this access behavior, it is possible to accurately determine the purchasing behavior of the accessed product 201.

Further, the purchasing behavior determination unit 13 determines the purchasing behavior based on the above access behavior and the product fluctuation information extracted in S3 (S4). Since the purchasing behavior is determined based on the actual fluctuation (increase / decrease in the number) of the accessed product 201, it is possible to reliably and accurately determine the purchasing behavior of the accessed product 201.

Further, the purchasing behavior determination unit 13 determines, as a purchasing behavior, one of an action of taking out the product 201 from the display place (for example, the product shelf 102), an action of returning the product 201 to the display place, and an action of contacting the product 201 (S4). ). Thereby, as a purchasing behavior, it is possible to clearly distinguish whether the customer P has taken the product 201, returned the product 201 to the original position, or just touched the product 201.

Further, the behavior analysis system 1 of the present embodiment includes the above-mentioned behavior analysis device 1 and at least one image pickup unit 2 for photographing the inside of the store 100 and acquiring an image. In this configuration, in order for the behavior analysis device 1 to determine the purchasing behavior of the customer P by using the image acquired by the image pickup unit 2 in the store 100, for example, an IC (integrated circuit) tag may be attached to the product 201. It is possible to determine the purchasing behavior without managing the product 201 by providing a sensor for each product shelf 102. Therefore, it is possible to reduce the cost of the entire system by eliminating the need to install an IC tag or a sensor.

<Embodiment 2>
The other embodiments of the present invention will be described below with reference to the drawings. In the present embodiment, in the configuration of the first embodiment, an example of further specifying for which product the purchasing behavior determined by the purchasing behavior determination unit 13 is performed is shown.

[Structure of behavior analysis system]
FIG. 8 is a block diagram schematically showing the configuration of the behavior analysis system 50 of the present embodiment. In the behavior analysis system 50 of the present embodiment, in the behavior analysis device 1 shown in the first embodiment, the storage unit 14 further includes a product information storage unit 14e and a purchase data storage unit 14f, and the behavior analysis The apparatus 1 has the same configuration as that of the first embodiment except that the list creating unit 20 is further provided. The list creation unit 20 is configured by, for example, a CPU separate from the control unit 19, but may be configured by the same CPU as the control unit 19. The details of the function of the list creation unit 20 will be described later in the operation explanation of the behavior analysis system 50.

The product information storage unit 14e stores the product information for identifying each product 201 displayed in the store 100 in association with the display location of each product 201. In the present embodiment, the correspondence between the product information of each product 201 and the display location is represented as a product map, and this product map is stored in the product information storage unit 14e. FIG. 9 schematically shows a product map 201M showing a correspondence relationship between the product information of each product 201 and the display location.

In the product map 201M, the product information a to p for specifying each product 201 is shown in association with the display location of each product 201. The product information a to p may be, for example, a specific name (product name) of the product 201, or may be a type of the product 201 (for example, a toothbrush, toothpaste, etc.). Of these, the product information a to d specifies each product 201 in rows A, B, C, and D of the product shelf 102 on the first stage from the top when the product shelf 102 has a four-stage configuration. Shows product information. Similarly, the product information e to h indicate the product information specifying each product 201 in the A column, the B column, the C column, and the D column of the product shelf 102 in the second row from the top, and the product information i to l are. Product information that identifies each product 201 in rows A, B, C, and D of the product shelf 102 in the third row from the top is shown, and the product information m to p are the product shelves 102 in the upper or fourth row. The product information that identifies each product 201 in columns A, B, C, and D is shown.

The purchase data storage unit 14f stores information (purchase data) regarding the product 201 actually purchased by the customer P in the store 100. As the above purchase data, for example, POS (Point of Sales) data can be considered. The POS data is data at the time of sale of the product 201, and includes information such as "when", "at which store", "which product", "how much", and "how many pieces were sold". For example, at the cash register in the store 100 (also simply called a cash register), POS data is acquired by scanning the barcode attached to the product 201 at the time of payment with a scanner, and this POS data is stored in the purchase data storage unit 14e in almost real time. Transferred and stored. The above POS data does not include "who" information, and even if the purchase data is leaked to the outside, the customer P (for example, personal name) who purchased the product may be identified. do not have.

For example, the purchase data acquired by the cache register is temporarily recorded on a recording medium (for example, an optical disk), and the purchase data is read by the reading unit 18 from the recording medium to acquire the purchase data. May be good. Further, the purchase data may be directly input by the input unit 15, or may be acquired by receiving the purchase data from an external terminal device (for example, a server) via the communication unit 17.

[Behavioral analysis system operation]
FIG. 10 is an explanatory diagram showing an example of a processing flow in the behavior analysis system 50 of the present embodiment. The behavior analysis method realized by the behavior analysis system 50 of the present embodiment includes a product information storage step (S1') before the image acquisition step (S1) of the first embodiment, and includes a purchase behavior determination step (S4). Is followed by a product identification step (S5), a list creation step (S6), and a list storage step (S7). Hereinafter, a part different from the first embodiment will be described.

(S1'; Product information storage process)
In S1', the product information (for example, the product map 201M in FIG. 9) associated with the display location of each product 201 is stored in the product information storage unit 14e in advance. For example, the system administrator operates the input unit 15 to directly input the product information, the storage medium storing the product information is read by the reading unit 18, and the product information is received from the outside via a communication line. The above product information can be stored in the product information storage unit 14e. The product information storage process of S1'may be performed at any timing as long as it is performed by the purchasing behavior determination process of S4.

(S5; product identification process)
After S4, the purchasing behavior determination unit 13 recognizes the product 201 whose display state has changed and the display location based on the product variation information extracted by the product variation information extraction unit 12. For example, when the product fluctuation information extraction unit 12 extracts product fluctuation information (information that the product 201 has decreased) for the product 201 in column A on the product shelf 102 in the first row from the top in FIG. 9, the purchasing behavior. Based on the above-mentioned product fluctuation information, the determination unit 13 recognizes the product 201 whose display state has changed and its display location (first row, row A). Then, the purchasing behavior determination unit 13 reads the product information (product information a in FIG. 9) of the product 201 corresponding to the recognized display location from the product information storage unit 14e, and based on the read product information, the purchasing behavior Specify the target product 201 (specify what the product 201 is). The specific result is stored in, for example, the determination result storage unit 14d of the storage unit 14.

(S6; list creation process, S7; list storage process)
In S6, the list creation unit 20 is a list of product information about the product 201 determined by the customer P to have taken from the display place as a purchasing behavior among the products 201 specified by the purchasing behavior determination unit 13 (hereinafter, simply ". Also called "product list"). FIG. 11 schematically shows an example of the product list L. In the product list L, the product information of the product 201 determined by the purchasing behavior determination unit 13 to be taken from the display place by the customer is summarized as a list for each customer. In S7, the product list L thus created is stored in the product information storage unit 14e of the storage unit 14.

After S7, for example, the control unit 19 transfers and stores the product information included in the created product list L and the product 201 to the purchase data storage unit 14f after the customer P actually purchases (settlement) the product 201. Compare with the data to determine if these data match. Then, in the case of matching (when the product included in the product list L and the product recorded as the purchase data match), the control unit 19 stores the product information and the purchase data in association with each other. It is stored in the unit 14 (product information storage unit 14e or purchase data storage unit 14f).

As described above, in the present embodiment, in S5, the purchasing behavior determination unit 13 identifies the product 201 that is the target of the purchasing behavior based on the product information read from the product information storage unit 14e, so that the customer P It is possible to know what the product 201 (the product taken from the display place by the customer P, the product returned to the display place, or the product just touched) was specifically.

Further, the list creation unit 20 creates the product list L and stores it in the product information storage unit 14e. For example, the system administrator reads the product list L from the product information storage unit 14e, and the customer P displays the display location. It is possible to clearly grasp the product 201 taken from.

<Programs and recording media>
The behavior analysis device 1 of the behavior analysis system 50 described in the above-described first and second embodiments can be configured by, for example, a computer (PC) in which a predetermined program (application software) is installed. By reading and executing the above program by a computer (for example, a control unit 19 as a CPU), each unit of the behavior analysis system 50 can be operated to execute each process (each step) described above. Such a program is acquired by downloading from the outside via a network, for example, and is stored in the program storage unit 14a. Further, the program is recorded on a computer-readable recording medium R such as a CD-ROM (Compact Disk-Read Only Memory), and the computer reads the program from the recording medium R and stores it in the program storage unit 14a. It may be in the form.

<Supplement>
In the above-described first and second embodiments, an example of determining the purchasing behavior of the customer P with respect to the product 201 displayed on the product shelf 102 has been described, but even when the product 201 is displayed without using the product shelf 102. , It is possible to determine the purchasing behavior by adopting the methods described in the first and second embodiments. For example, even if products (toilet paper, etc.) are piled up on the floor together with a cardboard box, or products (frozen food, etc.) are displayed in a large case with an open top surface, customer P. The purchasing behavior of the customer P can be determined based on the attitude information and the product fluctuation information of the customer P.

Further, in the first and second embodiments, the product fluctuation information is extracted by comparing the images before and after the customer P stays in front of the product shelf 102 (see FIGS. 6 and 7). It is not limited to this example. For example, the product variation information may be extracted by comparing the image when the customer P is staying in front of the product shelf 102 with the image after the customer P is staying in front of the product shelf 102. In this case, while the customer P is staying in front of the product shelf 102, the fluctuation of the product 201 hidden by the customer P cannot be extracted, but the product 201 not hidden by the customer P is described in 2 above. The fluctuation can be extracted by comparing the two images, which makes it possible to determine the purchasing behavior of the customer P.

<Others>
The behavioral analysis device, the behavioral analysis system, the behavioral analysis method, the program, and the recording medium described in each of the above embodiments may be expressed as follows.

1. 1. It is a behavior analysis device that analyzes customer behavior in the store.
A posture estimation unit that estimates the posture of the customer in the store and acquires posture information based on the image in the store, and a posture estimation unit.
A product variation information extraction unit that analyzes the display state of products in the store and extracts product variation information indicating changes in the display state from the image.
A behavior analysis device including a purchasing behavior determination unit that determines the purchasing behavior of the customer based on the posture information and the product fluctuation information.

2. 2. The product fluctuation information extraction unit determines whether or not the customer has stayed in front of the product based on the attitude information, and if it determines that the customer has stayed, the image of the frame before staying and the image of the frame before staying. The behavior analyzer according to 1 above, wherein the product fluctuation information is extracted by comparing the image of the frame after staying with the image of the frame and analyzing the display state of the product.

3. 3. The behavior analysis device according to 1 or 2, wherein the purchasing behavior determination unit detects the customer's access behavior to the product based on the change in the posture information.

4. 3. The behavior analysis device according to 3, wherein the purchasing behavior determination unit determines the purchasing behavior based on the access behavior of the customer to the product and the product fluctuation information.

5. The purchasing behavior determination unit is characterized in that, as the purchasing behavior, any one of an action of taking out the product from the display place, an action of returning the product to the display place, and an action of contacting the product is determined. The behavioral analyzer according to any one of 4.

6. Further provided with a product information storage unit that stores product information for identifying each product displayed in the store in association with the display location of each product.
The purchasing behavior determination unit recognizes the product whose display state has changed and the display location based on the product fluctuation information extracted by the product fluctuation information extraction unit, and also recognizes the product corresponding to the recognized display location. The behavior analysis according to any one of 1 to 5, wherein the product information is read from the product information storage unit, and the product that is the target of the purchasing behavior is specified based on the read product information. Device.

7. Among the products specified by the purchase behavior determination unit, a list creation unit for creating a list of the product information for the products determined to be taken from the display place by the customer as the purchase behavior is further provided.
6. The behavioral analysis device according to 6, wherein the product information storage unit further stores the list.

8. The behavioral analyzer according to any one of 1 to 7 above,
A behavior analysis system including an image pickup unit that photographs the inside of the store and acquires the image.

9. It is a behavior analysis method that analyzes customer behavior in the store.
A posture estimation process of estimating the posture of the customer in the store and acquiring posture information based on the image in the store, and
A product variation information extraction process that analyzes the display state of products in the store based on the image and extracts changes in the display state of the products as product variation information.
A behavior analysis method comprising a purchasing behavior determination process for determining a customer's purchasing behavior based on the attitude information and the product fluctuation information.

10. In the product variation information extraction step, it is determined whether or not the customer has stayed in front of the product based on the attitude information, and if it is determined that the customer has stayed, the image of the frame before staying and the image of the frame before staying. The behavior analysis method according to 9 above, wherein the product fluctuation information is extracted by comparing the image of the frame after staying with the image of the frame and analyzing the display state of the product.

11. 9. The behavior analysis method according to 9 or 10, wherein in the purchasing behavior determination step, the customer's access behavior to the product is detected based on the change in the posture information.

12. 11. The behavior analysis method according to 11 above, wherein in the purchasing behavior determination step, the purchasing behavior is determined based on the access behavior of the customer to the product and the product fluctuation information.

13. From 9 above, the purchasing behavior determination step is characterized in that, as the purchasing behavior, any one of an action of taking out the product from the display place, an action of returning the product to the display place, and an action of contacting the product is determined. The behavior analysis method according to any one of 12.

14. Based on the product fluctuation information extracted in the product fluctuation information extraction step, the product whose display state has changed and the display location are recognized, and the product information of the product corresponding to the recognized display location is stored in the product information storage unit. The behavior analysis method according to any one of 9 to 13, further comprising a product specifying step of specifying the product subject to the purchasing behavior based on the product information read from the product.

15. A list creation step of creating a list of the product information about the product determined to be taken from the display place by the customer as the purchasing behavior among the products specified in the product specifying process.
The behavior analysis method according to 14 above, further comprising a list storage step of storing the list in the product information storage unit.

16. A program that causes a computer to execute the behavioral analysis method according to any one of 9 to 15.

17. A computer-readable recording medium on which the program according to 16 is recorded.

Although the embodiments of the present invention have been described above, the scope of the present invention is not limited to this, and can be extended or modified without departing from the gist of the invention.

The present invention can be used in a system for analyzing customer behavior in a store.

1 Behavior analysis device 2 Imaging unit 11 Attitude estimation unit 12 Product fluctuation information extraction unit 13 Purchasing behavior judgment unit 14e Product information storage unit 14f Purchasing data storage unit 20 List creation unit 50 Behavior analysis system 100 Stores P Customer

Claims (21)

It is a behavior analysis device that analyzes customer behavior in the store.
A posture estimation unit that estimates the posture of the customer in the store and acquires posture information based on the image of the inside of the store.
A product variation information extraction unit that analyzes the display state of products in the store and extracts product variation information indicating changes in the display state from the image.
It is provided with a purchasing behavior determination unit that determines the purchasing behavior of the customer based on the posture information and the product fluctuation information.
The purchasing behavior determination unit detects the customer's access behavior to the product based on the change in the attitude information, and before and after the access behavior based on the product fluctuation information before and after the access behavior. A behavior analysis device, characterized in that the purchase behavior of the customer is determined by detecting the increase or decrease of the product . The purchase behavior determination unit is characterized in that, as the purchase behavior, any one of an action of taking out the product from the display place, an action of returning the product to the display place, and an action of contacting the product is determined. The behavioral analyzer described in. The purchasing behavior determination unit detects that the access behavior is once based on the change in the attitude information, and based on the product fluctuation information, the product is one before and after one access. The second aspect of claim 2, wherein when it is detected that the number of products has decreased, it is determined that the customer has taken one product from the display place in one access behavior as the purchasing behavior. Behavior analyzer. The purchasing behavior determination unit detects that the access behavior is twice based on the change in the attitude information, and based on the product fluctuation information, the product is two before and after the two accesses. The claim is characterized in that, when it is detected that the number of products has decreased, it is determined that the customer has taken the product one by one from the display place in each of the two access behaviors as the purchasing behavior. 2. The behavior analyzer according to 2. The purchasing behavior determination unit detects that the access behavior is once based on the change in the attitude information, and increases or decreases the product before and after the one access based on the product fluctuation information. When it is detected that there is no such product, it is determined that the purchase behavior is that the customer has only contacted the product in one access behavior and did not take the product. Item 2. The behavior analyzer according to Item 2. The purchasing behavior determination unit detects that the access behavior is twice based on the change in the attitude information, and increases or decreases the product before and after the two accesses based on the product fluctuation information. When it is detected that there is no such product, it is determined that the purchase action is to return the product taken by the customer in the first access action to the display place in the second access action. The behavioral analyzer according to claim 2, wherein the behavior analyzer is characterized. The product fluctuation information extraction unit determines whether or not the customer has stayed in front of the product based on the attitude information, and if it determines that the customer has stayed, the image of the frame before staying and the image of the frame before staying. The behavioral analyzer according to any one of claims 1 to 6, wherein the product fluctuation information is extracted by comparing with the image of the frame after staying and analyzing the display state of the product. .. Further provided with a product information storage unit that stores product information for identifying each product displayed in the store in association with the display location of each product.
The purchasing behavior determination unit recognizes a product whose display state has changed and a display location based on the product fluctuation information extracted by the product fluctuation information extraction unit, and also recognizes the product corresponding to the recognized display location. The action according to any one of claims 1 to 7, wherein the product information is read from the product information storage unit, and the product that is the target of the purchasing behavior is specified based on the read product information. Analysis equipment. Among the products specified by the purchase behavior determination unit, a list creation unit for creating a list of the product information for the products determined to be taken from the display place by the customer as the purchase behavior is further provided.
The behavioral analysis device according to claim 8, wherein the product information storage unit further stores the list. The behavioral analyzer according to any one of claims 1 to 9,
A behavior analysis system including an image pickup unit that photographs the inside of the store and acquires the image. It is a behavior analysis method that analyzes customer behavior in the store.
A posture estimation step in which the posture estimation unit estimates the posture of the customer in the store and acquires posture information based on the image taken by the image pickup unit in the store.
A product variation information extraction process in which the product variation information extraction unit analyzes the display state of the products in the store based on the image and extracts the variation of the display state of the products as the product variation information.
The purchasing behavior determination unit includes a purchasing behavior determination process for determining the purchasing behavior of the customer based on the attitude information and the product fluctuation information.
In the purchase behavior determination step, the purchase behavior determination unit detects the customer's access behavior to the product based on the change in the attitude information, and based on the product fluctuation information before and after the access behavior. , A behavior analysis method comprising detecting the purchase behavior of the customer by detecting the increase / decrease of the product before and after the access behavior. In the purchasing behavior determination step, the purchasing behavior determination unit determines, as the purchasing behavior, one of an action of taking out the product from the display place, an action of returning the product to the display place, and an action of contacting the product. 11. The behavior analysis method according to claim 11. In the purchase behavior determination step, the purchase behavior determination unit detects that the access behavior is once based on the change in the attitude information, and based on the product fluctuation information, the access is performed once. When it is detected that the number of the product has decreased by one before and after, it is determined that the customer has taken one product from the display place in one access behavior as the purchasing behavior. The behavior analysis method according to claim 12. In the purchase behavior determination step, the purchase behavior determination unit detects that the access behavior is twice based on the change in the attitude information, and based on the product fluctuation information, the access is performed twice. When it is detected that the number of the products has decreased by two before and after, it is determined that the customer has taken the product one by one from the display place in each of the two access behaviors as the purchasing behavior. The behavior analysis method according to claim 12, characterized in that. In the purchase behavior determination step, the purchase behavior determination unit detects that the access behavior is once based on the change in the attitude information, and based on the product fluctuation information, the access is performed once. When it is detected that there is no increase or decrease of the product before and after, it is determined that the purchase behavior is that the customer only touched the product in one access behavior and did not take the product. The behavior analysis method according to claim 12, wherein the behavioral analysis method is performed. In the purchase behavior determination step, the purchase behavior determination unit detects that the access behavior is twice based on the change in the attitude information, and based on the product fluctuation information, the access is performed twice. When it is detected that the product has not increased or decreased before and after, as the purchasing action, the action of returning the product taken by the customer in the first access action to the display place in the second access action is performed. The behavior analysis method according to claim 12, wherein it is determined that there was a problem. In the product variation information extraction step, the product variation information extraction unit determines whether or not the customer has stayed in front of the product based on the attitude information, and if it is determined that the customer has stayed, stays. Any of claims 11 to 16, wherein the product fluctuation information is extracted by comparing the image of the previous frame with the image of the frame after staying and analyzing the display state of the product. Behavior analysis method described in Crab. The purchasing behavior determination unit further includes a product specifying process for specifying the product that is the target of the purchasing behavior.
In the product specifying process, the purchasing behavior determination unit recognizes a product whose display state has changed and a display location based on the product variation information extracted by the product variation information extraction unit in the product variation information extraction step. At the same time, the product information corresponding to the recognized display location is read from the product information storage unit, and the product to be the target of the purchasing behavior is specified based on the read product information. The behavior analysis method according to any one of Items 11 to 17. The list creation unit creates a list of the product information about the products specified by the purchase behavior determination unit in the product identification process and which are determined by the customer to be taken from the display place as the purchase behavior. List creation process and
The behavior analysis method according to claim 18, wherein the product information storage unit further includes a list storage step for storing the list. A program that causes a computer to execute the behavioral analysis method according to any one of claims 11 to 19. A computer-readable recording medium on which the program according to claim 20 is recorded.

Images