KR102594256B1 - Method, program, and apparatus for monitoring behaviors based on artificial intelligence - Google Patents

Abstract

According to an embodiment of the present disclosure, an artificial intelligence-based behavior monitoring method, program, and apparatus are provided. The method comprises the steps of: synchronizing first image data captured in a first direction based on the face of a person subject to behavior monitoring, and second image data captured in a second direction different from the first direction; generating an analysis result for at least one detection item included in each of a plurality of detection targets based on each of the synchronized first image data and second image data by using a pre-trained neural network model; and estimating a behavior of the person by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data. Therefore, the method can complexly determine and accurately monitor what behavior the person takes in a specific environment based on various detection results.

Description

Artificial intelligence-based behavior monitoring method, program, and device {METHOD, PROGRAM, AND APPARATUS FOR MONITORING BEHAVIORS BASED ON ARTIFICIAL INTELLIGENCE}

This disclosure relates to data analysis technology, and specifically to methods and devices for estimating and monitoring based on complex judgment results about human behavior based on artificial intelligence.

Within an environment built for a specific purpose, situations arise where it is necessary to check the actions taken by a person and analyze the consequences of the actions taken by the person. For example, in an educational environment where exams are administered, there is a need to monitor what actions the test taker takes during exam time. In particular, unlike offline exams, it is difficult to effectively check the test taker's behavior and surrounding environment in online exams. Therefore, in an online exam environment, it is more important for administrators to accurately analyze the actions taken by test takers in real time to determine whether there has been any cheating.

As can be seen from the above examples, it is not easy to effectively monitor human behavior and the surrounding environment in an environment built online. Although there are conventional technologies that analyze specific human behavior using sensing devices such as cameras, most of them analyze specific human behavior based only on fragmentary information obtained in specific situations. However, if analysis is performed based on only fragmentary information like this, it cannot be accurately interpreted whether a person is taking an action that requires judgment in a specific environment. For example, when cheating is detected by analyzing only frontal images taken of people in an online exam environment, the limited information available from the frontal images makes it impossible to judge it as cheating or even if it is suspected to be cheating. Even if this is not the case, the probability of being misjudged as cheating increases.

Republic of Korea Patent Publication No. 10-2007-0050029 (May 14, 2007)

The present disclosure was developed in response to the above-described background technology, and seeks to provide a method and device that can complexly determine and accurately monitor what actions a person takes within a specific environment based on various detection results.

However, the problems to be solved by this disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood based on the description below.

An artificial intelligence-based behavior monitoring method performed by a computing device is disclosed according to an embodiment of the present disclosure for realizing the above-described task. The method synchronizes first image data captured in a first direction based on the face of a person subject to behavior monitoring, and second image data captured in a second direction different from the first direction. step; Using a pre-trained neural network model, generating an analysis result for at least one sensed item included in each of a plurality of sensed objects based on each of the synchronized first and second image data; and estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data.

Alternatively, the sensed item may be status information identified based on the subclass of the sensed object. And, the status information may be changeable depending on the person's behavior.

Alternatively, the plurality of sensing objects may include, in addition to body parts of the person, at least one of an object other than the person, a sound of an object associated with the person's action, or a time of an object associated with the person's action. You can.

Alternatively, the neural network model may include: a first model that tracks a person's gaze based on image data captured in the first direction; and a second model that estimates the pose of the person based on image data captured in the second direction.

Alternatively, using the pre-trained neural network model, generate an analysis result for at least one detection item included in each of the plurality of detection objects based on each of the synchronized first image data and second image data. In the step of inputting the synchronized second image data into the second model, if the extracted feature points for the human face and the feature points for the human body are misaligned by more than a predetermined angle, the synchronized first video data It may include correcting the feature points extracted from the synchronized second image data based on the coordinate system of the feature points extracted by inputting them into the first model.

Alternatively, determining a judgment condition for estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data comprises: , determining whether a first expected behavior according to the first analysis result and a second expected behavior according to the second analysis result match; estimating reliability of at least one of the first analysis result or the second analysis result; and estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability.

Alternatively, the step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability may include the first expected behavior and the first expected behavior. 2 If the expected behavior matches, a step of estimating the matched expected behavior as the person's behavior may be included.

Alternatively, the step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability may include the first expected behavior and the first expected behavior. 2 If the expected behavior is inconsistent and the reliability of the second analysis result is less than a threshold, the method may include estimating the first expected behavior as the person's behavior.

Alternatively, the step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability may include the first expected behavior and the first expected behavior. 2 If the expected behavior is inconsistent and the reliability of the second analysis result is more than a threshold, estimating the person's behavior based on the judgment condition derived by combining the first analysis result and the second analysis result It can be included.

Alternatively, the first direction may be a frontal direction of the face, and the second direction may be a side direction of the face.

According to an embodiment of the present disclosure for realizing the above-described object, a computer program stored in a computer-readable storage medium is disclosed. When the computer program runs on one or more processors, it performs operations for monitoring behavior based on artificial intelligence. At this time, the operations include synchronizing first image data captured in a first direction with respect to the face of the person subject to behavior monitoring, and second image data captured in a second direction different from the first direction. ; Using a pre-trained neural network model, generating an analysis result for at least one detection item included in each of a plurality of detection objects based on each of the synchronized first image data and the second image data; and an operation of estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data.

According to an embodiment of the present disclosure for realizing the above-described task, a computing device for monitoring behavior based on artificial intelligence is disclosed. The device includes a processor including at least one core; a memory containing program codes executable on the processor; And it may include a network unit for acquiring image data. At this time, the processor synchronizes first image data captured in a first direction and second image data captured in a second direction different from the first direction based on the face of the person subject to behavior monitoring, Using a pre-trained neural network model, an analysis result for at least one detection item included in each of a plurality of detection objects is generated based on each of the synchronized first image data and the second image data, and the synchronized The person's behavior can be estimated by combining a first analysis result based on the first image data and a second analysis result based on the synchronized second image data.

The present disclosure can provide a method and device that can complexly determine and accurately monitor what actions a person takes within a specific environment based on various detection results.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.
FIG. 2 is a block diagram illustrating a process for monitoring behavior of a computing device according to an embodiment of the present disclosure.
Figure 3 is a conceptual diagram detailing the estimation process for each behavior of a computing device according to an embodiment of the present disclosure.
Figure 4 is a conceptual diagram showing a computational process for feature correction of a neural network model according to an embodiment of the present disclosure.
Figure 5 is a flowchart showing an artificial intelligence-based behavior monitoring method according to an embodiment of the present disclosure.
Figure 6 is a flowchart showing a method for monitoring behavior in an online test environment according to an embodiment of the present disclosure.

Below, with reference to the attached drawings, embodiments of the present disclosure are described in detail so that those skilled in the art (hereinafter referred to as skilled in the art) can easily implement the present disclosure. The embodiments presented in this disclosure are provided to enable any person skilled in the art to use or practice the subject matter of this disclosure. Accordingly, various modifications to the embodiments of the present disclosure will be apparent to those skilled in the art. That is, the present disclosure can be implemented in various different forms and is not limited to the following embodiments.

The same or similar reference numerals refer to the same or similar elements throughout the specification of this disclosure. Additionally, in order to clearly describe the present disclosure, reference numerals in the drawings may be omitted for parts that are not related to the description of the present disclosure.

As used in this disclosure, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “X uses A or B” should be understood to mean one of natural implicit substitutions. For example, unless otherwise specified in the present disclosure or the meaning is not clear from the context, “X uses A or B” means that It can be interpreted as one of the cases where all B is used.

The term “and/or” as used in this disclosure should be understood to refer to and include all possible combinations of one or more of the listed related concepts.

The terms “comprise” and/or “comprising” as used in this disclosure should be understood to mean that certain features and/or elements are present. However, the terms "comprise" and/or "including" should be understood as not excluding the presence or addition of one or more other features, other components, and/or combinations thereof.

Unless otherwise specified in this disclosure or the context is clear to indicate a singular form, the singular should generally be construed to include “one or more.”

The term “Nth (N is a natural number)” used in the present disclosure can be understood as an expression used to distinguish the components of the present disclosure according to a predetermined standard such as a functional perspective, a structural perspective, or explanatory convenience. there is. For example, in the present disclosure, components performing different functional roles may be distinguished as first components or second components. However, components that are substantially the same within the technical spirit of the present disclosure but must be distinguished for convenience of explanation may also be distinguished as first components or second components.

The term "acquisition" used in this disclosure refers to generating or receiving data in an on-device form, as well as receiving data through a wireless communication network with an external device or system. It can be understood that

Meanwhile, the term "module" or "unit" used in this disclosure refers to a computer-related entity, firmware, software or part thereof, hardware or part thereof. , can be understood as a term referring to an independent functional unit that processes computing resources, such as a combination of software and hardware. At this time, the “module” or “unit” may be a unit composed of a single element, or may be a unit expressed as a combination or set of multiple elements. For example, a "module" or "part" in the narrow sense is a hardware element of a computing device, or set of pieces thereof, an application program that performs a specific function of software, a process implemented through software execution, or a program execution. It may refer to a set of instructions for . Additionally, as a broad concept, “module” or “unit” may refer to the computing device itself constituting the system, or an application running on the computing device. However, since the above-described concept is only an example, the concept of “module” or “unit” may be defined in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

As used in this disclosure, the term "model" refers to a system implemented using mathematical concepts and language to solve a specific problem, a set of software units to solve a specific problem, or a process to solve a specific problem. It can be understood as an abstract model of a process. For example, a deep learning “model” can refer to an overall system implemented as a neural network that has the ability to solve problems through learning. At this time, the neural network can have problem-solving capabilities by optimizing parameters connecting nodes or neurons through learning. A deep learning “model” may include a single neural network or a set of neural networks that are a combination of multiple neural networks.

The term “image” used in this disclosure may refer to multidimensional data composed of discrete image elements. In other words, “image” can be understood as a term referring to a digital representation of an object that can be seen by the human eye. For example, “image” may refer to multidimensional data consisting of elements corresponding to pixels in a two-dimensional image. “Image” may refer to multidimensional data consisting of elements corresponding to voxels in a three-dimensional image.

The explanation of the foregoing terms is intended to aid understanding of the present disclosure. Therefore, if the above-mentioned terms are not explicitly described as limiting the content of the present disclosure, it should be noted that the content of the present disclosure is not used in the sense of limiting the technical idea.

1 is a block diagram of a computing device according to an embodiment of the present disclosure.

The computing device 100 according to an embodiment of the present disclosure may be a hardware device or part of a hardware device that performs comprehensive processing and calculation of data, or may be a software-based computing environment connected to a communication network. For example, the computing device 100 may be a server that performs intensive data processing functions and shares resources, or it may be a client that shares resources through interaction with the server. Additionally, the computing device 100 may be a cloud system that allows a plurality of servers and clients to interact and comprehensively process data. Since the above description is only an example related to the type of computing device 100, the type of computing device 100 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

Referring to FIG. 1, a computing device 100 according to an embodiment of the present disclosure may include a processor 110, a memory 120, and a network unit 130. there is. However, since FIG. 1 is only an example, the computing device 100 may include other components for implementing a computing environment. Additionally, only some of the configurations disclosed above may be included in computing device 100.

The processor 110 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for performing computing operations. For example, the processor 110 may read a computer program and perform data processing for machine learning. The processor 110 may process computational processes such as processing input data for machine learning, extracting features for machine learning, and calculating errors based on backpropagation. The processor 110 for performing such data processing includes a central processing unit (CPU), a general purpose graphics processing unit (GPGPU), a tensor processing unit (TPU), and a custom processing unit (TPU). It may include a semiconductor (ASIC: application specific integrated circuit), or a field programmable gate array (FPGA: field programmable gate array). Since the type of processor 110 described above is only an example, the type of processor 110 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The processor 110 may perform preprocessing on a plurality of image data obtained for behavior monitoring. Even when shooting the same object or environment, the information expressed in the image data may vary depending on the shooting time, shooting direction, etc. Accordingly, the processor 110, which utilizes a plurality of image data for behavior monitoring, may perform synchronization to ensure accuracy of complex analysis and judgment of the plurality of image data. At this time, synchronization can be understood as an operation to adjust or match the shooting timing of a plurality of image data based on the object that is a detection target in a plurality of image data in which the same environment is captured. For example, unlike image data in which a person's face is captured from the front, image data in which a person's face is captured from the side may not include information about the face depending on the movement of the person's head. Accordingly, the processor 110 may perform preprocessing to synchronize the shooting timing between the two data so that the image data captured from the front and the image data captured from the side can be used complementary to each other. Through this synchronization preprocessor, the processor 110 can stably combine analysis results of two image data and improve the accuracy of behavior estimation results for behavior monitoring.

The processor 110 may use a pre-trained neural network model to generate analysis results for each of a plurality of detection objects based on image data captured by a person subject to behavior monitoring. At this time, the detection object can be understood as a component of image data that serves as a standard for estimating human behavior. Additionally, the analysis result of the detection object may be information indicating what action a person takes based on the detection object present in the image data. Specifically, the detection target may be any one of a person's body part, an object other than a person, the sound of an object associated with a person's action, or the time of an object associated with a person's action. Objects related to a person's behavior may be a part of the person's body, or they may be objects that can be changed by being influenced by the person's behavior. Additionally, the analysis result of the detection object may be information about actions performed by a person based on the sound of a person's body part, object, or object present in the image data, or the time of the object. That is, the processor 110 inputs image data into a pre-trained neural network model and analyzes the results of each detection target present in the image data to detect a specific human action performed under a specific environment for behavioral monitoring. It can be created from basic data.

The processor 110 may estimate a specific human behavior for behavior monitoring based on analysis results for each of a plurality of image data performed based on the above-described neural network model. The processor 110 may combine the analysis results of the neural network model for each of the plurality of synchronized image data to interpret what actions the person subject to behavioral monitoring has taken. At this time, the processor 110 may use a ruleset to estimate human behavior based on the combined analysis results. A rule set may be a set of behavior classes that are candidates for detection in a specific environment for behavior monitoring and judgment conditions for each behavior class. Additionally, rulesets can be created, changed, or modified by administrators who have established a specific environment for behavior monitoring. In other words, the processor 110 comprehensively determines the analysis results for each detection object present in the video data based on a rule set that can be customized to suit a specific environment for behavior monitoring and determines the human behavior present in the video data. can be estimated. For example, assuming that the environment for behavior monitoring is an environment for an online exam, the ruleset created by the test proctor's client may be used to detect exam cheating and/or that may be suspected of cheating, but not cheating. It may be a set of judgment conditions for each abnormal behavior, misconduct, and/or abnormal behavior. The processor 110 may combine the analysis results of the neural network model for each of the frontal image data and the side image data, and identify a combination result that matches the judgment conditions included in the above-described rule set. At this time, the combination of analysis results is understood as a computational process that compares the behavior expected from the analysis result of the frontal image data with the behavior expected from the analysis result of the side image data, and combines the analysis results into one ground action based on the comparison result. It can be. In addition, the processor 110 may determine whether the test taker has performed a cheating act or an abnormal behavior included in the rule set under a situation confirmed by observation data, based on the combination result matching the judgment conditions.

In this way, the processor 110 derives information about the basic behavior to determine what action the person will take based on the expected behavior based on each of the plurality of image data, and based on the rule set created according to the specific environment. You can monitor specific human behavior. Accordingly, the processor 110 can perform precise, accurate, and stable detection and analysis compared to detection and analysis based on single image data.

The memory 120 according to an embodiment of the present disclosure may be understood as a structural unit including hardware and/or software for storing and managing data processed in the computing device 100. That is, the memory 120 can store any type of data generated or determined by the processor 110 and any type of data received by the network unit 130. For example, the memory 120 may be a flash memory type, hard disk type, multimedia card micro type, card type memory, or random access memory (RAM). ), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory), magnetic memory , a magnetic disk, or an optical disk may include at least one type of storage medium. Additionally, the memory 120 may include a database system that controls and manages data in a predetermined system. Since the type of memory 120 described above is only an example, the type of memory 120 may be configured in various ways within a range understandable to those skilled in the art based on the contents of the present disclosure.

The memory 120 can manage data necessary for the processor 110 to perform operations, a combination of data, and program code executable on the processor 110 by structuring and organizing them. For example, the memory 120 may store image data acquired through the network unit 130, which will be described later. The memory 120 includes program codes that operate the processor 110 to learn a deep learning model, program codes that operate the processor 110 to estimate human behavior using the learned deep learning model, and program codes that are executed. As a result, various data calculated can be stored.

The network unit 130 according to an embodiment of the present disclosure may be understood as a structural unit that transmits and receives data through any type of known wired or wireless communication system. For example, the network unit 130 is a local area network (LAN), wideband code division multiple access (WCDMA), long term evolution (LTE), and WiBro (wireless). broadband internet, 5th generation mobile communication (5G), ultra wide-band wireless communication, ZigBee, radio frequency (RF) communication, wireless LAN, wireless fidelity ), data transmission and reception can be performed using a wired or wireless communication system such as near field communication (NFC), or Bluetooth. Since the above-described communication systems are only examples, the wired and wireless communication systems for data transmission and reception of the network unit 130 may be applied in various ways other than the above-described examples.

The network unit 130 may receive data necessary for the processor 110 to perform operations through wired or wireless communication with any system, server, or client. Additionally, the network unit 130 may transmit data generated through calculations of the processor 110 through wired or wireless communication with any system, server, or client. For example, the network unit 130 may receive video data of a person subject to behavior monitoring through wired or wireless communication with a sensing device such as a camera or a client equipped with a sensing device. Additionally, the network unit 130 may receive user input through a user interface implemented in a sensing device or a client equipped with a sensing device. The network unit 130 may transmit various data generated through calculations of the processor 110 based on image data through wired or wireless communication with a sensing device or a client equipped with a sensing device.

FIG. 2 is a block diagram illustrating a process for monitoring behavior of a computing device according to an embodiment of the present disclosure. And, Figure 3 is a conceptual diagram detailing the estimation process for each behavior of a computing device according to an embodiment of the present disclosure.

Referring to FIG. 2 , the computing device 100 according to an embodiment of the present disclosure may perform preprocessing on a plurality of image data about a person subject to behavior monitoring. At this time, the image data includes first image data 11 captured in a first direction based on the face of the person subject to behavior monitoring, and second image data 13 captured in a second direction different from the first direction. may include. That is, the computing device 100 may synchronize the first image data 11 captured in the first direction with respect to the face and the second image data 13 captured in the second direction different from the first direction.

For example, the first image data 11 and the second image data 13 may be images captured through a camera installed in a space for online testing. At this time, a sensing device such as a camera may be a component of the client owned by the test taker. The monitoring device can be installed in the testing space to individually photograph the test taker's face from the front and side. When the online test starts, the sensing device included in the test taker's client generates first image data 11, in which the test taker's face is captured from the front, and second image data 13, in which the test taker's face is captured from the side. can be created. The computing device 100 may acquire first image data 11 and second image data 13 generated by the test taker's client through wired or wireless communication with the test taker's client. In addition, the computing device 100 uses both the first image data 11 and the second image data 13 for analysis by the neural network model 200, which will be described later. Video data (13) can be synchronized. Since the second image data 13 is captured based on the side of the person, the person's face may not be expressed properly depending on the person's movement or shooting angle. Therefore, in order to accurately and stably perform analysis on the face included in the body part of the person being detected, the computing device 100 can synchronize the first image data 11 and the second image data 13. .

The computing device 100 may input the synchronized first image data and the synchronized second image data into the neural network model 200. At this time, the neural network model 200 tracks the person's gaze based on image data captured in the first direction and the pose of the person based on the image data captured in the second direction. It may include a second model 220 for estimation. That is, the computing device 100 can input synchronized first image data to the first model 210 optimized for the task of eye tracking. Additionally, the computing device 100 may input synchronized second image data to the second model 220 optimized for the task of pose estimation.

For example, the first model 210 may receive image data captured from the front based on the person's face and track the person's gaze present in the image data. Specifically, the first model 210 may generate a crop image by extracting the face area of the person from image data in which the person is photographed. The first model 210 can recognize human eyes by extracting features based on the cropped image. Additionally, the first model 210 can track the person's gaze by analyzing the movements and changes of the pupils included in the recognized eyes. For this eye tracking, the first model 210 may include a neural network optimized for image processing. Additionally, the first model 210 may be learned based on not only supervised learning but also semi-supervised learning, unsupervised learning, and self-supervised learning.

The second model 220 may receive image data captured from the side based on the person's face and detect the pose taken by the person present in the image data. Specifically, the second model 220 may receive image data and classify the body part and the background based on a plurality of feature points for identifying a person's pose, thereby generating a mask for the body part. Additionally, the second model 220 can estimate what pose the person is taking by analyzing the mask for the body part. For such pose estimation, the second model 220 may include a neural network optimized for image processing. Additionally, the second model 220 may be learned based on not only supervised learning but also semi-supervised learning, unsupervised learning, and self-supervised learning.

The computing device 100 may generate an analysis result for at least one sensed item included in each of the plurality of sensed objects through the neural network model 200 to which a plurality of synchronized image data is input. The computing device 100 may generate analysis results for each sensed item through the first model 210 into which the synchronized first image data is input. The computing device 100 may generate analysis results for each sensed item through the second model 220 into which the synchronized second image data is input. At this time, the detection items of the first model 210 and the detection items of the second model 220 may be the same or different.

For example, if the detection target is a human body part, subclasses of the detection target can be divided into face, arm, etc. The face can be further divided into eyes, nose, mouth, and ears. And, the arm can be further divided into hand, palm, and fingers. Detection items are status information that can be detected based on each subclass of the detection target, and may include gaze direction, speech status, hand position, palm direction, etc. In other words, a detection item can represent a specific state or appearance that can appear when a subclass of the detection object moves or changes depending on a person's actions.

In other words, the computing device 100 may input the synchronized first image data into the pre-trained first model 210 to generate a first analysis result 15 including analysis results for various sensed items. . Additionally, the computing device 100 may input the synchronized second image data into the pre-trained second model 220 to generate a second analysis result 17 including analysis results for various sensed items. At this time, each of the first analysis result 15 and the second analysis result 17 may include analysis results matching each of the detection items such as gaze direction, speech status, hand position, palm direction, etc.

For example, assuming that the environment for behavior monitoring is an environment for an online test, the analysis result matching the gaze direction can indicate the detection result of whether the test taker is looking into the display that checks the test paper. . The analysis result matching whether the test taker made a speech may represent the result of detecting whether the shape of the test taker's mouth has changed. The analysis result matching the hand position may indicate the result of detecting whether the test taker's left or right hand is moving within a reference space determined according to the test taker's body and the arrangement of the desk. Analysis results matching the above-described detection items may be included in the first analysis result 15 and the second analysis result 17, respectively. In this way, the computing device 100 may individually generate analysis results for at least one sensing item included in each of the plurality of sensing objects using the first model 210 and the second model 220. Through this process, the computing device 100 can obtain various information to be used in calculations for behavior estimation, which will be described later, from image data captured at various angles.

Referring to FIG. 2 , the computing device 100 may combine a plurality of analysis results generated through the neural network model 200 to estimate the behavior of a person to be monitored. In order to generate the behavior estimation result 19, the computing device 100 uses the first analysis result 15 generated through the first model 210 and the second analysis result generated through the second model 220 ( 17) can be combined. At this time, in order to combine the first analysis result 15 and the second analysis result 17, the computing device 100 uses the first expected behavior and the second analysis result 19 inferred from the first analysis result 15. The second expected behavior inferred from can be used. In addition, the computing device 100 may utilize the reliability of each of the first analysis result 15 and the second analysis result 17 in order to combine the first analysis result 15 and the second analysis result 17. .

Specifically, the computing device 100 may estimate the first expected behavior according to the first analysis result 15 and the second expected behavior according to the second analysis result 17 based on a predetermined rule set. The computing device 100 may determine whether the estimated first expected behavior matches the estimated second expected behavior. Additionally, the computing device 100 analyzes at least one of the first model 210 or the second model 220 to determine at least one of the reliability of the first analysis result 15 or the reliability of the second analysis result 17. It can be estimated. The computing device 100 generates a first analysis result ( By combining 15) and the second analysis result (17), basis information corresponding to the judgment conditions for estimating behavior can be generated.

For example, referring to FIG. 3, if the first analysis result 15 is derived as “the test taker is looking outside the display checking the test paper” based on the detection item called gaze direction, the computing device (100 ) can estimate the first expected behavior as cheating from the first analysis result (15) based on a predetermined rule set. That is, the computing device 100 may confirm that the first analysis result 15 matches the cheating judgment conditions included in the predetermined rule set and estimate the first expected behavior as cheating. If the second analysis result 17 is derived as "the lower end of the test taker's elbow is out of the reference area" based on the detection item called body part, the computing device 100 determines the second analysis result based on a predetermined rule set. From (17), the second expected behavior can be estimated as abnormal behavior. That is, the computing device 100 may confirm that the second analysis result 17 matches the abnormal behavior judgment conditions included in the predetermined rule set and estimate the second expected behavior as abnormal behavior. In this case, when the first expected behavior and the second expected behavior are inconsistent, the computing device 100 determines the first analysis result as “the lower part of the test taker’s elbow is outside the reference area, and the test taker is looking outside the display.” (15) and the second analysis result (17) can be combined. At this time, the results of the above-described combination may vary depending on the reliability of the first analysis result 15 and the second analysis result 17. For example, if the reliability of the second analysis result 17 is less than the threshold, the computing device 100 may derive the first analysis result, “Test taker is looking outside the display,” as a combination result. If the reliability of the second analysis result 17 is greater than or equal to the threshold, the computing device 100 determines the result of the combination as "the lower part of the test taker's elbow is outside the reference area, and the test taker is gazing out of the display" as described above. It can be derived. However, the computing device 100 controls the accuracy of the first analysis result 15 and the accuracy of the second analysis result 17 so that the first analysis result 15 is reflected more heavily in the final behavior estimation. The accuracy of the combination result can also be derived based on the weighted sum. In addition, the computing device 100 combines the results based on the weighted sum of the correlation between the first analysis result 15 and the cheating or abnormal behavior and the correlation between the second analysis result 17 and the cheating or abnormal behavior. Correlations between cheating or abnormal behavior can also be derived. At this time, correlation can be understood as a quantitative indicator that indicates the extent to which a specific analysis result influences the judgment of misconduct or abnormal behavior. In other words, the content of the combination result is derived from the above-mentioned "the bottom of the test taker's elbow is outside the reference area, and the test taker is gazing outside the display," while the accuracy or correlation of the combination result is as described above. The results of the weighted agreement can be reflected. In addition, the accuracy or correlation of the combination result may be reflected in the calculation process for estimating the final behavior of the computing device 100. In this way, the computing device 100 can increase the accuracy of behavior estimation by using a complementary combination of analysis results of data taken in different directions (or angles) to monitor behavior.

The computing device 100 may use a predetermined rule set tailored to a specific environment for behavior monitoring in order to derive the behavior estimation result 19. Specifically, the computing device 100 may identify a judgment condition for each action class included in a predetermined rule set that matches the combination result of the first analysis result 15 and the second analysis result 17. At this time, when the first analysis result 15 and the second analysis result 17 are combined through a weighted sum, the computing device 100 combines the first analysis result 15 and the second analysis result 17 By filtering the judgment conditions for each behavior class included in the predetermined rule set, reflecting the accuracy or correlation, the behavior estimation result (19) corresponding to the behavior class can be derived. The computing device 100 may estimate the behavior of a person whose behavior class corresponding to the identified judgment condition must be detected for behavior monitoring under a specific environment.

For example, assuming that the environment for behavior monitoring is an environment for an online test, the computing device 100 screens a predetermined rule set to obtain the first analysis result 15 and the second analysis result 17. The judgment conditions of the first behavior class regarding misconduct or the judgment conditions of the second behavior class regarding abnormal behavior that match the combination result of can be identified. The computing device 100 may derive a first behavior class related to misconduct or a second behavior class related to abnormal behavior corresponding to the identified judgment conditions as the behavior estimation result 19.

Figure 4 is a conceptual diagram showing a computational process for feature correction of a neural network model according to an embodiment of the present disclosure.

Referring to FIG. 4, the computing device 100 according to an embodiment of the present disclosure may input the synchronized first image data 21 into the first model to extract feature points for each of the human face and body. . The computing device 100 may input the synchronized second image data 25 into the second model and extract feature points for each of the person's face and body. At this time, if the feature points for the human face and the feature points for the human body extracted from the synchronized second image data 25 are offset by more than a predetermined angle, the computing device 100 generates the synchronized first image data 21. Based on the feature points extracted from , the feature points extracted from the synchronized second image data 25 can be corrected.

For example, assume that the first synchronized image data 21 is an image captured from the frontal direction of the person, and the second synchronized image data 25 is an image captured from the side direction of the person. Unlike the first image data 21, where face capture is less affected by human movement, face capture in the second image data 25 is greatly affected by human movement. In other words, when the person moves, such as when the angle of the person's face rotates more than 45 degrees relative to the body, the second model cannot properly extract feature points for the person's face from the second image data 25 and performs unstable analysis. It will be performed. Therefore, when the feature points for the face and the feature points for the body extracted from the second image data 25 are offset by more than 45 degrees, the computing device 100 extracts the feature points from the first image data 21, which have little effect on the person's movement. The coordinates of the feature points extracted from the second image data 25 can be adjusted based on the coordinate system 30 of the feature points. The computing device 100 may generate an output of the second model using the feature points of the second image data 25 whose coordinates have been adjusted. That is, the second model can generate analysis results for each detected item using feature points corrected based on feature points extracted by the first model. Through such correction, the computing device 100 can secure the stability of data analysis.

Figure 5 is a flowchart showing an artificial intelligence-based behavior monitoring method according to an embodiment of the present disclosure.

Referring to FIG. 5, the computing device 100 according to an embodiment of the present disclosure includes first image data captured in a first direction with respect to the face of a person subject to behavior monitoring, and image data different from the first direction. The second image data captured in the second direction can be synchronized (S110). For example, the computing device 100 may synchronize first image data generated through a camera installed in front of a person's face and second image data generated through a camera installed on a side of the person's face. At this time, synchronization of the first image data and the second image data can be understood as a task of adjusting the time when the person's face is photographed to match. The computing device 100 can effectively use the first image data and the second image data together as data for behavior estimation through synchronization.

The computing device 100 uses a pre-trained neural network model to detect at least one detection item included in each of the plurality of detection objects based on each of the first image data and the second image data synchronized through step S110. Analysis results can be generated (S120). At this time, the plurality of detection objects may include at least one of a body part of a person, an object other than the person, a sound of an object related to the person's action, or a time of an object related to the person's action. Additionally, the detection item is status information identified based on the subclass of the detection target, and may be status information that can change depending on human behavior. For example, the computing device 100 inputs first image data synchronized with second image data into a first model that tracks gaze based on image data taken from the front of the face to identify body parts, objects, and objects. An analysis result for at least one detection item included in at least one of sound and object time may be generated. The computing device 100 inputs second image data synchronized with the first image data into a second model that estimates the pose of a person based on image data taken from the side of the face, and generates body parts, objects, sounds of objects, and An analysis result may be generated for at least one detection item included in at least one of the times of the object. The detection target and detection item that serve as analysis standards for each of the first model and the second model may be the same or different.

The computing device 100 may estimate human behavior by combining the first analysis result generated through step S120 and the second analysis result generated through step S120 (S130). Specifically, the computing device 100 may determine whether the first expected behavior according to the first analysis result and the second expected behavior according to the second analysis result match. At this time, the expected behavior may correspond to the estimated behavior included in a predetermined rule set according to the behavior monitoring environment. Additionally, the computing device 100 may estimate the reliability of at least one of the first analysis result or the second analysis result. The computing device 100 may estimate the person's behavior by combining the first analysis result and the second analysis result based on the previously determined match and the estimated reliability. For example, when the first expected behavior and the second expected behavior match, the computing device 100 may estimate the matched expected behavior as human behavior. If the first expected behavior and the second expected behavior are inconsistent, and the reliability of the second analysis result is less than a threshold, the computing device 100 may estimate the first expected behavior as the person's behavior. If the first expected behavior and the second expected behavior are inconsistent and the reliability of the second analysis result is greater than or equal to the threshold, the computing device 100 combines the first analysis result and the second analysis result based on the judgment condition derived, Human behavior can be estimated. At this time, the person's behavior estimated based on each expected behavior or judgment condition may be determined based on a predetermined rule set according to the behavior monitoring environment. At this time, the threshold may be a value predetermined by the administrator for behavior monitoring.

Figure 6 is a flowchart showing a method for monitoring behavior in an online test environment according to an embodiment of the present disclosure.

Referring to FIG. 6, the computing device 100 according to an embodiment of the present disclosure may create an online exam based on a user request input through an online exam organizer client (S210). At this time, the environmental conditions for the online test, the rule set for monitoring the test taker's behavior, etc. may be determined by reflecting the user request entered through the host client. For example, the computing device 100 may determine an acquisition cycle for observation data, a rule set including definitions and judgment conditions for misconduct or abnormal behavior, etc., based on a user request input through the host client. After the rule set is created by a user request, it may be dynamically updated while the computing device 100 repeatedly performs behavior estimation.

When an online test is created (S210), the computing device 100 may acquire images of the test taker at a predetermined cycle (S220). For example, the computing device 100 may acquire frontal and side images of the test taker's face at intervals of 100 ms to 1 s through wired or wireless communication with a sensing device installed in the test taking space. At this time, the sensing device may be a component provided in the test taker's client or may be a component of the computing device 100. In addition, the acquisition cycle of observation data can be predetermined according to the environmental conditions of the online test through step S210.

The computing device 100 may synchronize the front image and the side image (S230). Even if the front image and the side image are images created by shooting the same subject (i.e. the test taker), the computing device 100 adjusts the shooting timing of the front image and the side image based on the test taker's face for stable analysis. Preprocessing operations can be performed.

The computing device 100 may perform analysis for each detected item by inputting the synchronized front image 51 and the synchronized side image 52 into a neural network model (S240). The computing device 100 may input the synchronized frontal image 51 into the pre-trained first model 210 to generate a first analysis result 53 for the detected item to which the first model 210 matches. there is. The computing device 100 may input the synchronized side image 52 into the pre-trained second model 220 to generate a second analysis result 54 for the detected item to which the second model 220 matches. there is. The computing device 100 may estimate the first expected action 55 based on the first analysis result 53. The first expected behavior (55) can be understood as a behavior determined when cheating or abnormal behavior is expected based on the first analysis result (53). The computing device 100 may estimate the second expected action 56 based on the second analysis result 54. The second expected behavior (56) can be understood as a behavior determined when cheating or abnormal behavior is expected based on the second analysis result (54).

The computing device 100 may determine whether the first expected behavior 55 and the second expected behavior 56 match (S251). If the first expected behavior 55 and the second expected behavior 56 match, the computing device 100 may determine the test taker's behavior based on the matched expected behavior (S255). If the first expected action 55 and the second expected action 56 do not match, the computing device 100 may determine whether the reliability of the second analysis result 54 is greater than or equal to the threshold (S261). If the reliability of the second analysis result 54 is less than the threshold, the computing device 100 may determine the test taker's behavior using the first expected behavior 55 (S265). If the reliability of the second analysis result 54 is greater than or equal to the threshold, the computing device 100 may combine the first analysis result 53 and the second analysis result 54 (S270). And, the computing device 100 may estimate the test taker's behavior based on the combination result (S280).

For example, if the first expected behavior 55 and the second expected behavior 56 match cheating or abnormal behavior, the computing device 100 may determine the test taker's behavior based on the matched behavior. If the first expected behavior 55 is inconsistent with cheating and the second expected behavior 56 is abnormal behavior, the computing device 100 determines whether the reliability of the second analysis result 54 is above the threshold. You can. If the reliability of the second analysis result 54 is less than the threshold, the computing device 100 may determine the first expected behavior 55, which is cheating, as the behavior of the test taker. When the reliability of the second analysis result 54 is greater than or equal to the threshold, the computing device 100 combines the first analysis result 53 and the second analysis result 54 and determines whether the combination result exists in a predetermined rule set. It can be figured out. If the computing device 100 determines that the result of the combination exists as a judgment condition included in a predetermined rule set, the computing device 100 may determine the test taker's behavior to be cheating or abnormal behavior, which is behavior according to the identified judgment condition.

The various embodiments of the present disclosure described above may be combined with additional embodiments and may be changed within the scope understandable to those skilled in the art in light of the above detailed description. The embodiments of the present disclosure should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form. Accordingly, all changes or modified forms derived from the meaning and scope of the claims of the present disclosure and their equivalent concepts should be construed as being included in the scope of the present disclosure.

Claims (12)

An artificial intelligence-based behavior monitoring method performed by a computing device including at least one processor, comprising:
Synchronizing first image data captured in a first direction with respect to the face of a person subject to behavior monitoring, and second image data captured in a second direction different from the first direction;
Using a pre-trained neural network model, generating an analysis result for at least one sensed item included in each of a plurality of sensed objects based on each of the synchronized first and second image data; and
Combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data to estimate the person's behavior; including,
The step of determining a judgment condition for estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data, comprising:
determining whether a first expected behavior according to the first analysis result and a second expected behavior according to the second analysis result match;
estimating reliability of at least one of the first analysis result or the second analysis result; and
combining the first analysis result and the second analysis result based on the determined match and the estimated reliability to estimate the person's behavior;
Including,
method.
According to claim 1,
The detection items are:
It is status information identified based on the subclass of the detection target,
The status information is,
that can change depending on the person's actions,
method.
According to claim 1,
The plurality of detection targets are,
In addition to the body part of the person, it includes at least one of an object other than the person, a sound of an object associated with the action of the person, or a time of an object associated with the action of the person.
method.
According to claim 1,
The neural network model is,
a first model that tracks a person's gaze based on image data captured in the first direction; and
a second model that estimates a pose of a person based on image data captured in the second direction;
Including,
method.
According to claim 4,
Generating an analysis result for at least one detection item included in each of a plurality of detection objects based on each of the synchronized first and second image data using the pre-trained neural network model, comprising:
When the feature points of the human face and the feature points of the human body extracted by inputting the synchronized second image data into the second model are offset by more than a predetermined angle,
Inputting the synchronized first image data into the first model and correcting feature points extracted from the synchronized second image data based on a coordinate system of the extracted feature points;
Including,
method.
delete According to claim 1,
The step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability includes,
If the first expected behavior and the second expected behavior match,
estimating the matched expected behavior as the person's behavior;
Including,
method.
According to claim 1,
The step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability includes,
When the first expected behavior and the second expected behavior are inconsistent, and the reliability of the second analysis result is less than a threshold value,
estimating the first expected behavior as the person's behavior;
Including,
method.
According to claim 1,
The step of estimating the person's behavior by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability includes,
When the first expected behavior and the second expected behavior are inconsistent, and the reliability of the second analysis result is greater than or equal to a threshold value,
estimating the person's behavior based on judgment conditions derived by combining the first analysis result and the second analysis result;
Including,
method.
According to claim 1,
The first direction is,
It is in the frontal direction of the face,
The second direction is,
In the side direction of the face,
method.
A computer program stored in a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs operations for monitoring behavior based on artificial intelligence,
The above operations are:
An operation of synchronizing first image data captured in a first direction with respect to the face of a person subject to behavior monitoring, and second image data captured in a second direction different from the first direction;
Using a pre-trained neural network model, generating an analysis result for at least one detection item included in each of a plurality of detection objects based on each of the synchronized first image data and the second image data; and
An operation of estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data,
The operation of determining a judgment condition for estimating the person's behavior by combining a first analysis result based on the synchronized first image data and a second analysis result based on the synchronized second image data includes:
Determine whether the first expected behavior according to the first analysis result and the second expected behavior according to the second analysis result match, and estimate the reliability of at least one of the first analysis result or the second analysis result, Estimating the person's behavior based on a judgment condition derived by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability,
computer program.
A computing device for monitoring behavior based on artificial intelligence,
A processor including at least one core;
a memory containing program codes executable on the processor; and
A network unit for acquiring image data;
Including,
The processor,
Synchronizing first image data captured in a first direction and second image data captured in a second direction different from the first direction based on the face of the person subject to behavioral monitoring,
Using a pre-trained neural network model, generate an analysis result for at least one detection item included in each of a plurality of detection objects based on each of the synchronized first and second image data,
Determine whether a first expected behavior according to a first analysis result based on the synchronized first image data matches a second expected behavior according to a second analysis result based on the synchronized second image data, and perform the first analysis Estimating the reliability of at least one of the result or the second analysis result, based on a judgment condition derived by combining the first analysis result and the second analysis result based on the determined match and the estimated reliability, estimating the behavior of said person,
Device.

Images