KR20230169880A - Cathexis learning system and method using AI in an untact learning based on Web browser - Google Patents

Abstract

A system and method for attention learning in web browser-based untact learning are disclosed. The system for attention learning in web browser-based untact learning includes: a learning content server providing untact lecture learning content; and a user terminal accessing the learning content server to drive an educational content viewer, which is installed therein such that artificial intelligence attention learning with learning content is performed using AI face recognition/motion recognition and sound reproduction technology of front camera image data of a user terminal in untact learning. In the untact learning, the educational content viewer of the user terminal includes a media playback part and a facial contour recognition part module; in order to induce a user to learn with attention about untact lecture learning content focused on a camera of the user terminal connected to the learning content server (ubcloud server) while keeping an eye on the content, an alarm/warning message is issued to the corresponding user terminal using sound and messaging technology when the eyes of the user are found out of the content at a predetermined angle through face contour recognition based on contours of the face and five feature points of the face: two eyes/one nose/two ears; and web browser-based attention learning using ubcloud artificial intelligence is provided in untact learning.

Description

Attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning {Cathexis learning system and method using AI in an untact learning based on Web browser}

The present invention relates to an attention learning system and method using artificial intelligence in non-face-to-face learning (ubcloud). More specifically, in non-face-to-face learning of online lectures, the educational content viewer of the user terminal is a media player and a facial contour recognition module. Equipped with a face outline and 5 facial feature points of eyes 2/nose/ears 2 so that the user can look directly at the learning content focused on the front camera of the user terminal linked to the learning content server (ubcloud server) and focus attention. Through facial contour recognition, if the gaze looking at the user's terminal at a certain angle deviates, an alarm/warning message is generated to the user's terminal using sound and messaging technology, and in non-face-to-face learning, the web browser-based ubcloud artificial intelligence is used. This relates to an attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning (ubcloud) that provides attention learning.

Face recognition technology mainly uses appearance-based matching method and feature-based face recognition. Face recognition is recognized differently depending on the camera's shooting angle, lighting direction, pose, changes in expression, and changes in the face over time.

Feature-based face recognition uses a detection method using Haar-like features and a detection method using MCT (Modified Census Transform) images from image data captured with a digital camera, IoT device camera, or smartphone camera. . Using a face and eye detector learned with Haar-like features from the input image of the smartphone camera, the facial outline and forehead/eyes/nose/mouth feature points are detected, and the region of interest (ROI) is used to detect circular pupils. The eye area set as (Region of Interest) is converted to grayscale, and the histogram of the eye image [the pixel of each pixel on the value, the number of corresponding pixel values on the y-axis], binarize the image of the eye, and perform preprocessing of the photo of the eye area through histogram equalization. In the face area, eyebrows, eyes, nose, The face is recognized by detecting face data with facial feature points of the mouth and outline, extracting texture features and shape features, and comparing the similarity with feature points of the face photo stored in the face recognition DB.

The feature values of the eyebrows, eyes, nose, mouth, and chin in the face area are expressed as the difference between the sum of pixels included in the white area of the Haar-like feature and the sum of the pixels included in the black area.

For example, the distance from the eye area detected in a face area photo of standard width and height size to both endpoints of the right and left eyes, and the size of the pupil (iris) extracted using the Hough circle transform algorithm. The value is used as the feature value.

Figure 1a is a configuration diagram of an existing face recognition device.

The face recognition device 100 is an image display device, an image capture device, a face recognition server, a tablet PC, a laptop, a personal PC, a smartphone, a personal digital assistant (PDA), a mobile communication terminal, and It may be one of intelligent robots, etc.

The face recognition device 100 includes an input image acquisition unit 112 that acquires an input image from a camera; A front pose image is generated by detecting the face area in the input image and normalizing the face pose, and the perspective of the front pose image is used to remove perspective distortion depending on the distance between the camera and the subject. ) a normalization unit 114 that normalizes and generates a normalized image; a feature vector extraction unit 116 that extracts a feature vector representing the face of the subject from the normalized image; and a face recognition unit 118 that recognizes the face of the subject included in the input image by applying the feature vector to a previously learned classification model.

The input image acquisition unit 112 acquires an input image from a camera. The cameras may be depth perception cameras, stereo cameras, and color cameras (eg, Kinect cameras, etc.). Additionally, the input image is an image containing the face of a subject to be recognized and includes a two-dimensional still image and a moving image. Input images may include color images, depth images, and color-depth (RGB-D) images.

The normalization unit 114 detects the face area from the input image and normalizes the face pose and perspective to generate a normalized image. If there is a change in the face pose, the gray scale, shape, location of feature points, etc. change, so the face recognition rate deteriorates. Additionally, if the distance between the camera and the subject changes, even if it is the same subject, different perspective distortions occur at each photographed location, making it appear as if a different subject was photographed. Therefore, it is necessary to normalize the facial pose and perspective of the input image to improve the face recognition rate.

The normalization unit 114 inputs facial images for learning in various poses to the input layer of the first artificial neural network, and trains the first artificial neural network so that facial images for learning in frontal poses are output from the output layer of the first artificial neural network. Face pose normalization learning unit; And inputting data output from the output layer of the first artificial neural network to the input layer of the second artificial neural network, and training the second artificial neural network so that a facial image for learning without perspective distortion is output from the output layer of the second artificial neural network. Includes a perspective normalization learning unit.

The normalization unit inputs learning facial images of various poses with various perspective distortions to the input layer of an integrated artificial neural network that integrates the first artificial neural network and the second artificial neural network for which learning has been completed, and has no perspective distortion of the frontal pose. The integrated artificial neural network is trained so that facial images for learning are output from the output layer of the integrated artificial neural network.

The feature vector extraction unit 116 is determined through machine learning and extracts a feature vector representing the subject's face from the normalized image.

A feature vector is a vector whose elements include feature values used for face recognition. Filters used to extract feature vectors include Gabor filter, Haar filter, LBP (Local Binary Pattern) - including DLBP (Discriminative LBP), ULBP (Uniform LBP), NLBP (Number LBP) - etc., but are necessarily limited to these. does not work and other filters can be used.

The face recognition unit 118 recognizes the face of the subject included in the input image by applying the feature vector extracted by the feature vector extraction unit 116 to the previously learned classification model. Pre-trained classification models may include, but are not necessarily limited to, Support Vector Machine (SVM), Linear Discriminant Analysis (LDA), and Softmax.

The virtual face image generator 124 may generate a plurality of virtual face images that are used for learning by the normalizer 114, the feature vector extractor 116, and the face recognition unit 118.

The plurality of virtual face images refers to face images generated by the virtual face image generator 124 transforming a 3D face model synthesized using one or more 2D reference images obtained from a camera.

As prior art 1 related to this, “artificial intelligence deep learning-based video recognition system and method” is registered in patent registration number 10-2103521.

Figure 1b is a configuration diagram of a conventional artificial intelligence deep learning-based video recognition system. Through deep learning of artificial intelligence, various images are pre-learned, and the pre-learning results are reflected to analyze images on a frame-by-frame basis for distributed videos. In some cases, audio information from the videos is also analyzed to organize them in chronological order. We provide a video recognition system and method based on artificial intelligence deep learning that derives keywords and compares them with time-series keywords from video scripts accumulated in advance.

The system implemented by artificial intelligence includes a dataset learning unit that performs deep learning on multiple object images using keywords; DB unit that extracts keywords from video scripts, lists and stores them in time series; A video analysis unit that extracts keywords from objects in the image and the relationships between the objects through frame-by-frame image analysis and lists them in time series; and a comparison judgment unit that determines similarity by comparing keywords of the DB unit and the video analysis unit.

However, existing e-learning or u-learning systems simply provide online learning content, and use facial recognition technology and sound and messaging technology to generate an alarm when the learner inadvertently deviates from the facial gaze and enable focused learning. did not provide

Patent registration number 10-2103521 (registration date April 16, 2020), "Artificial intelligence deep learning-based video recognition system and method", Sangmyung University Industry-Academic Cooperation Foundation

The purpose of the present invention to solve the above problem is that in non-face-to-face learning of online lectures, the educational content viewer of the user terminal is equipped with a media playback unit and a facial outline recognition module, and the user terminal linked to the learning content server (ubcloud server) In order to focus your attention on the learning content focused on the front camera and focus your attention on the learning content in non-face-to-face lectures, your gaze is directed at a certain angle through facial outline recognition on a 5-point scale with facial features of eyes 2, nose, and ears 2. In case of a miss, an alarm/warning message is generated to the user terminal using sound and messaging technology, and artificial intelligence is used in web browser-based non-face-to-face learning, which provides attention learning using web browser-based ubcloud artificial intelligence in non-face-to-face learning. Provides an attention learning system using intelligence.

Another object of the present invention is to provide an attention learning method using artificial intelligence in web browser-based non-face-to-face learning.

In order to achieve the purpose of the present invention, an attention learning system using artificial intelligence in web browser-based non-face-to-face learning includes a learning content server that provides non-face-to-face online lecture learning content; Connects to the learning content server through a wired or wireless communication network to run an educational content viewer, and in non-face-to-face online learning, AI facial recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal is used to focus the learning content with artificial intelligence. A user terminal with an educational content viewer installed to learn; and an instructor's terminal,

The instructor's terminal transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server,

The user terminal includes a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes facial behavior patterns in the front camera image of the user terminal and recognizes a 5-point scale of facial features of eyes 2/nose/ears 2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end or the learner's voice is deviated from the screen by a certain angle or more. When recognized, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the learner's user terminal to enable focused learning, and the user terminal further includes a voice recognition module that recognizes the learner's voice,

The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.

The educational content viewer is equipped with a VOD media player, a chat screen, and an online learning memo space,

In non-face-to-face online learning, the instructor's terminal transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server.

A 5-point scale facial feature point of eyes 2/nose/ear 2 is used to recognize the facial contours of eyes 2/nose/ear 2 so that the non-face-to-face online lecture learning contents focused on the front camera of the user terminal can be focused on. In some cases, alarm/warning messages are generated using sound and messaging technology.

In order to achieve another object of the present invention, the attention learning method in non-face-to-face learning based on a web browser includes the steps of (a) registering and storing learner information and a standard-sized frontal face photo from a user terminal by a learning content server; (b) the learning content server issuing a QR code corresponding to user information and a frontal face photo; (c) A user terminal equipped with an educational content viewer and a facial recognition module uses the front-facing face recognition algorithm of the front-facing camera image to receive the recognition result of the front-facing face photo from the camera to the learning content server, along with pre-registered learner information and standards. Comparing the size of the frontal face photo and the facial features to confirm the learner's identity; (d) providing learning content from the learning content server to a user terminal of a pre-registered member; and (e) the learning content server analyzes the online learner's face image detected by the camera on each learner's user terminal and sends it to the corresponding user terminal for attention-focused learning when the gaze deviates from a certain angle or when the learner's voice is recognized. Including steps for generating a warning message or alarm,

In non-face-to-face online learning, the educational content viewer installed on the user terminal is equipped with a VOD media player, a chat screen, and an online learning memo space, and the instructor's terminal is connected to a plurality of learner terminals in a 1:N manner through the learning content server. It further includes the step of transmitting and receiving question and answer chat data,

The user terminal includes a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes facial behavior patterns in the front camera image of the user terminal and recognizes facial feature points on a 5-point scale of eyes 2/nose/ears 2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end or the learner's voice is deviated from the screen by a certain angle or more. When recognized, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the learner's user terminal to enable focused learning, and the user terminal further includes a voice recognition module that recognizes the learner's voice,

The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.

In the attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning of the present invention, the educational content viewer of the user terminal in non-face-to-face learning is equipped with a media playback unit and a facial outline recognition unit module, and a learning content server ( Facial outline recognition is performed on a 5-point scale of facial contours and facial features of eyes 2, nose, and ears 2 so that you can focus your attention on the non-face-to-face lecture learning content focused on the front camera of the user terminal linked to the ubcloud server. If the gaze is diverted at a certain angle, an alarm/warning message is generated on the user's terminal using sound and messaging technology, and it has the effect of providing attention learning using web browser-based ubcloud artificial intelligence in non-face-to-face learning.

Figure 1a is a configuration diagram of an existing face recognition device.
Figure 1b is a configuration diagram of a conventional artificial intelligence deep learning-based video recognition system.
Figures 2a and 2b are examples of implementation of an attention learning system using web browser-based ubcloud artificial intelligence in non-face-to-face online learning.
Figure 3a is a diagram showing the concept of a facial contour recognition artificial intelligence platform equipped with a tablet PC, smartphone, and PC-based learning content server during online learning.
Figure 3b is a diagram showing the face recognition function on the ubiquitous NSDAI platform.
Figures 3c and 3d show the goals and User Experience (1. User registration -> 2. Learning -> 3. QR) of the UBI cloud App and NS facere platform used on tablet PCs, smartphones, and PCs during non-face-to-face online learning and testing. Code creation -> 4. UBT App/Web login (QR code/passwd, ID/passwd) -> 5. Supervisor confirmation and test -> 6.7 Facial outline recognition of face by UBT App/Web camera 8. End of test) It's a picture.
Figure 4 shows the process of the test taker department from learner/test taker registration, 1) test taker registration, 2) learning, 3) QR code issuance, 4) QR code and face recognition, 5) supervisor confirmation (test taker face/test taker information), 6) Test taking - This is a picture showing the process.
Figure 5 is a screen of the candidate-face matching code issuance unit - candidate registration/learning/QR code issuance during the online test/UBT test based on the machine learning and learning results of the registered candidate's face.
Figure 6 is a screen of the test taker confirmation section in the UBT system, which measures the distance and similarity of facial feature points in a standard-sized frontal facial photo based on the vertex of the nose through AI-based facial contour recognition of the test taker's face.
Figure 7 shows the rule user definition section and program test screen when making a learner decision.
Figure 8 shows learning content server access/login during non-face-to-face online learning/left - QR code recognition / greeting heard when QR code is recognized (TTS) / right - face recognition (smartphone/tablet PC front camera) / face in the background This is a demonstration screen that includes the area recognition start/recognition rate display process - the UBT authentication process customization section utilizing the generated user code.
Figure 9 is a diagram showing the configuration of an attention learning system using artificial intelligence in non-face-to-face learning based on a tablet PC, smartphone, and PC-based web browser using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.
Figure 10 is a diagram explaining the function of the AI-based facial contour recognition module during online learning and testing.
Figure 11 is a flowchart showing a tablet PC, smartphone, and PC-based attention learning method using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

Hereinafter, the configuration and operation of the preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, if it is determined that a detailed description of a related known technology or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the attached drawing numbers are assigned the same drawing numbers in other drawings when indicating the same configuration.

In the attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning of the present invention, the educational content viewer of the user terminal in non-face-to-face learning is equipped with a media playback unit and a facial outline recognition unit module, and a learning content server ( Facial outline recognition is performed on a 5-point scale of facial contours and facial features of eyes 2, nose, and ears 2 so that students can focus their attention on non-face-to-face lecture learning content focused on the front camera of the user terminal linked to the ubcloud server. If the gaze strays at a certain angle, an alarm/warning message is generated to the user's terminal using sound and messaging technology, and it provides attention learning using web browser-based ubcloud artificial intelligence in non-face-to-face learning.

* Same facial feature point and outline recognition technology (ubcloud)

* Learning content server and test taker functions added

* The feature is that it is browser-based and can be used both online (read online and save online based on learning data) and offline (built-in library browser and web page, and store recognition data on the user's device).

* Use sound and messaging to focus attention

The attention learning system and method using artificial intelligence in web browser-based non-face-to-face learning of the present invention is an educational content viewer that is linked to a learning content server in a user terminal (tablet PC, smartphone, PC) during non-face-to-face online learning. is installed, and uses AI-based facial recognition (posenet algorithm, machine learning model which allows for real-time face pose estimation) technology using the front-facing camera of the user terminal using AI facial recognition/motion recognition/sound and messaging technology. It is equipped with a facial recognition module that recognizes the outline of the face and the characteristic points of the face on a 5-point scale of eyes2/nose/ears2, and learns from frontal facial photos taken using the front-facing camera (C) of the candidate's terminal when recognizing the face. Detects the learner's attendance by comparing it with the facial feature points of the learner's frontal face photo of a standard size in the content server's face photo DB, and in non-face-to-face online learning, uses AI-based real-time video data from the front camera of the user's terminal. By recognizing the facial contour and detecting the behavior patterns of the learner's face in the video, the face is focused on the front camera of the user terminal linked to the learning content server (ubcloud server) so that the user can look directly into the learning content and focus attention on the face. The outline of the face and facial features of eyes 2/nose/ear 2 are measured on a 5-point scale. If the gaze deviates at a certain angle through facial outline recognition, an alarm/warning message is generated on the learner's user terminal, and sound and messaging technology is used. So that you can focus your attention.

User terminals include laptops with Internet access, as well as smartphones and tablet PCs that receive online learning content from a learning content server through a wired or wireless communication network.

The user terminal is installed with an educational content viewer that is linked to the learning content server, and uses AI facial recognition/motion recognition/sound and messaging technology to identify the outline of the learner's face and facial features on a 5-point scale of eyes2, nose, and ears2. It is equipped with a facial recognition module that recognizes the face.

Figures 2a and 2b are examples of implementation of an attention learning system using web browser-based ubcloud artificial intelligence in non-face-to-face online learning.

Figure 3a is a diagram showing the concept of a facial contour recognition artificial intelligence platform equipped with a tablet PC, smartphone, and PC-based learning content server during online learning.

Figure 3b is a diagram showing the face recognition function on the ubiquitous NSDAI platform.

Figures 3c and 3d show the goals and User Experience (1. User registration->2. Learning->3. QR) of the UBI cloud App and NS facere platform used on tablet PCs, smartphones, and PCs during non-face-to-face online learning and testing. Code creation -> 4. UBT App/Web login (QR code/passwd, ID/passwd) -> 5. Supervisor confirmation and test -> 6.7 Facial outline recognition of face by UBT App/Web camera 8. End of test) It's a picture.

Figure 4 shows the process of the test taker department from learner/test taker registration, 1) test taker registration, 2) learning, 3) QR code issuance, 4) QR code and face recognition, 5) supervisor confirmation (test taker face/test taker information), 6) Test taking - This is a picture showing the process.

Figure 5 is a screen of the candidate-face matching code issuance unit - candidate registration/learning/QR code issuance during the online test/UBT test based on the machine learning and learning results of the registered candidate's face.

Figure 6 is a screen of the test taker confirmation section in the UBT system, which measures the distance and similarity of facial feature points in a standard-sized frontal facial photo based on the vertex of the nose through AI-based facial contour recognition of the test taker's face.

Figure 7 shows the rule user definition section and program test screen when making a learner decision.

Figure 8 shows learning content server access/login during non-face-to-face online learning/left - QR code recognition / greeting heard when QR code is recognized (TTS) / right - face recognition (smartphone/tablet PC front camera) / face in the background This is a demonstration screen that includes the area recognition start/recognition rate display process - the UBT authentication process customization section utilizing the generated user code.

(1) The user terminal is installed with an educational content viewer, and the educational content viewer is equipped with a 5-point scale facial recognition module that recognizes the facial features of eyes 2/nose/ear 2 in the learner's face image using artificial intelligence facial recognition. .

QR code was used as a recognition code for online learning and testing.

In the embodiment, the user terminal used a tablet PC, and the recognition code used a QR code.

Artificial intelligence that has learned the registered photo of the learner/test taker generates a user-specific recognition code (QR code), and the test taker displays the test ticket with the corresponding recognition code (QR code) attached to the camera of the test site PC or tablet PC. The user-recognized AI module learns the applicant information and a frontal face photo (a standard-size frontal face photo with the nose at its peak) and learns the applicant information and standard-size frontal face photo already registered in the face database of the content server. The resulting models are compared, the similarity of the facial features is calculated, and if it exceeds a certain value, it is recognized as the person himself and moves to the learning screen, where the learner begins non-face-to-face online learning. If the similarity of facial features is below a certain level (completely below, judgment on hold), information is transmitted to the supervisor's terminal through the server, and the supervisor conducts online learning after confirming the actual face and personal information.

<System configuration>

1) Non-face-to-face online learning: Candidate terminal (PC/smartphone/tablet) user program -> wired and wireless communication network (WAN, LTE 4G/5G) and internal network (LAN, Wi-Fi) -> learning content server (learning curriculum information) /Learner information) transmits the frontal face photo and learner information of each user terminal to determine whether the supervisor terminal will participate in learning.

2) When used offline: The artificial intelligence facial recognition module is installed in the user program (PC/smartphone/tablet), recognizes the facial outline of the candidate's face, and sends only the final facial recognition result to the learning content server.

<Partial APK of app prototype on learner terminal - attached for Android>

https://we.tl/t-wFdXexsors File download password: nsdevil

Install the app on your Android smartphone or tablet PC

<Web version function address>

1) Facial recognition module: https://facere.nsdai.org (id: nsdevil, passwd: nsdevil) > It works by taking a picture of the QR code with a smartphone and touching it to the recognition screen of the PC camera.

2) Facial contour recognition module of facial contour and 5-point scale facial feature points: https://headpos.ublrandd.com.np (id: nsdevil, passwd: nsdevil), eyes 2/nose/ear 2 of the learner's face Facial recognition of facial features on a 5-point scale

3) Voice recognition module

Speech recognition is a man-machine interface technology that removes noise from an acoustic speech signal input through a microphone, extracts the characteristics of the speech signal, and converts it into a set of words or text of a sentence (mapping). The process is stored in microphone -> AMP -> LPF -> ADC -> voice database.

The speech recognition module uses vector quantization, dynamic time warping (DTW), neural network, and HMM (Hidden Markov Model). method is used. The voice recognition module used a voice recognition module with modular hardware and software and a voice recognition API of STT (Speech To Text) technology.

Figure 9 is a diagram showing the configuration of an attention learning system using artificial intelligence in non-face-to-face learning based on a tablet PC, smartphone, and PC-based web browser using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

Figure 10 is a diagram explaining the function of the AI-based facial contour recognition module during online learning and testing.

The learning content server 100 having a test management unit includes a WWW server 101, a control unit 102, a member registration unit 103, a user authentication unit 104, a learning content provision unit 105, and a QR code management unit 106. , a supervisor confirmation unit, a face recognition unit (107), a test management unit (109), a test information DB (120), an applicant DB (121), and a face photo DB (123).

The attention learning system in non-face-to-face learning based on a web browser of the present invention is

A learning content server 100 that provides non-face-to-face online lecture learning content authored by an authoring tool;

The learning content server 100 is connected to the learning content server 100 through a wired or wireless communication network to run an educational content viewer, and in non-face-to-face online learning, the learning content is artificially created using AI facial recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal. User terminals (300, 310, 311) installed with an educational content viewer for intelligent attention learning; and an instructor's terminal,

The instructor's terminal transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server 100,

The user terminals 300, 310, and 311 include a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes facial behavior patterns in the front camera image of the user terminal and recognizes facial outlines and 5-point scales of facial features of eyes2/nose/ears2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are closer to the 2 points of the ears at either end or the learner's user is deviating from the screen by a certain angle or more When the terminal recognizes the learner's voice, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the learner's user terminal so that the learner's voice can be focused on learning, and the user terminal has an STT (STT) that recognizes the learner's voice as a microphone. It further includes a voice recognition module using Speech To Text technology,

The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.

To ensure that you can focus your attention on non-face-to-face online lecture learning content focused on the front camera of the user terminal, the facial outline and facial features of eyes 2, nose, and ears 2 are placed on a 5-point scale, and the gaze is set at a certain angle through facial recognition. In case of a miss or if the learner's voice is recognized on the learner's user terminal, an alarm/warning message is generated to the user terminal using sound and messaging technology to ensure focused learning.

The educational content viewer is equipped with a VOD media player, a chat screen, and an online learning memo space. In non-face-to-face online learning, the instructor's terminal (supervisor's terminal) connects multiple learners in a 1:N manner through the learning content server 100. Question and answer chat data is transmitted and received through terminals.

During non-face-to-face online learning, the learning content server 100 allows the instructor to provide learning content regardless of time and place and underline the online learning memo space (shared workspace) of the educational content viewer or add text, pictures, and photos. , Provides sending/receiving files with attached reports/learning files and sending/receiving notes to the instructor's terminal (supervisor terminal) and multiple learner terminals according to the instructor's floor control through voice recording,

During a non-face-to-face lecture, the instructor's terminal (supervisor's terminal) writes letters/graphics/memos in the shared workspace of the educational content viewer with multiple learner terminals in a 1:N manner through the learning content server 100. Write is displayed.

In non-face-to-face online learning, the instructor's terminal (supervisor terminal) transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server 100.

In non-face-to-face online learning, the instructor provides learning content during online learning through a learning content server regardless of time and place and underlines text, pictures, and photos in the online learning memo space (shared workspace) of the educational content viewer. , Through voice recording, it provides sending/receiving files with attached reports/learning files and sending/receiving notes to the instructor's terminal (supervisor terminal) and multiple learner terminals according to the instructor's floor control.

In non-face-to-face online learning, during a non-face-to-face lecture, the instructor's terminal (supervisor terminal) is connected to the shared workspace of the educational content viewer with a plurality of learner terminals in a 1:N manner through the learning content server 100. Text/graphic handwriting/memo writing is displayed.

The learning content server 100 provides a function to recognize a QR code as a recognition code when using a smartphone.

The user terminal uses any one of a tablet PC, a smartphone, and a PC equipped with a front-facing camera, and the educational content viewer downloaded from the learning content server 100 is installed, and the educational content viewer is VOD media. A chat screen is provided with the player.

The user terminal is

VOD media player that plays non-face-to-face lecture learning content;

A chat module that transmits and receives chat data between instructors and learners; and

A facial recognition module that recognizes the facial behavior pattern of the front camera image of the user terminal and recognizes the facial outline and facial feature points on a 5-point scale of eyes2/nose/ears2; and

If the gaze deviates from the screen by more than a certain angle (if the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end), the learner's user It includes a learner alarm generator that generates an alarm or outputs a warning message through the terminal.

The facial recognition module of the user terminal is an AI-based facial outline recognition technology that uses the posenet algorithm.

The facial recognition module uses a facial recognition algorithm to extract facial objects from a standard size corrected size/rotation/angle corrected frontal face photo of the learner during non-face-to-face online learning, and recognizes facial behavior patterns to identify the learner's face. Extract the outline and the facial feature points of Eye2/Nose/Ear2, and the Euclidean distance (pupil) of the facial feature points of Eye2/Nose/Ear2 with the center point (pupil) of the left/right ear and left/right eye, respectively ( Calculate the similarity with d), check whether the gaze is deviated by comparing it with specific points of the user's face photo in the face photo DB, and depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at both ends, the right/right Detects head movement to the left and detects facial patterns related to gaze deviation. When recognizing a face, if the facial outline is not recognized or if it deviates from the screen by more than a certain angle (the 3 points of the eyes/nose are close to the 2 points of the ears at both ends) (If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard value), facial behavior patterns captured by the camera of the user terminal are recognized, and the right eye and right ear are detected according to the angle at which the frontal face is turned to the left and right. Since the distance and the distance between the left eye and the left ear are different, an alarm is generated on the learner's user terminal or a warning message is output to encourage attention and learning.

The learning content server 100

WWW server 101;

A control unit 102 that controls provision of online learning content to the user terminal;

A member registration unit (103) connected to the control unit (102), which receives member information and stores and manages ID/Passwd;

A user authentication unit 104 connected to the control unit 102 and authenticating using QR code/Passwd, ID/Passwd, or personal certificate;

A QR code management unit 106 connected to the control unit 102 and issuing and managing a QR code corresponding to user information;

a learning content provider 105 connected to the control unit 102 and providing the online learning content to the user terminal;

A chat server 111 connected to the control unit 102 and transmitting and receiving chat data between the instructor and the learner;

It is connected to the control unit 102, and receives the frontal face photo of the learner/candidate from the camera from the user terminal, converts it into a standard size frontal face photo through standard size correction/rotation/angle correction, and stores it in the server's database. A proctor verification unit that verifies online learning/online exam qualifications by comparing the stored user information with a standard-sized frontal face photo and confirming it with the proctor at the proctor terminal;

It is connected to the control unit 102, and uses a facial recognition module in the user terminal to recognize the outline of the learner's face and facial feature points of eyes 2/nose/ear 2, and receives the face recognition results from the user terminal. Wealth (107); and

Test information DB that stores test papers, written answers, and scoring results of user terminals; It includes a candidate DB and a face DB that store candidate information and standard-sized frontal face photos.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a chat server 111 that transmits and receives chat data between the instructor and the learner.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a message sending/receiving service unit that transmits and receives message data to which report files of instructors and learners can be attached.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes an FTP server for transmitting and receiving learning data files.

Additionally, when taking an online test or UBT test, a test program (App) and test papers are provided to the test taker's terminal and the proctor's terminal, and the test taker's information, on-site face photo, and proctor information are managed. After storing the answers written on the test paper on each test taker's terminal within a certain test time, they are sent to the test server at the end of the test, and further include a test management unit that stores and manages the test takers' answers written on the test paper, scoring results, supervisor information, and test taker status information. do.

Additionally, the AI-based facial recognition technology used in the facial recognition module of the test taker's terminal and the anti-cheating module that utilizes a 5-point scale for eye2/nose/ear2 facial features recognizes the facial outline and facial feature points in the camera image of the tablet PC. To do this, the posenet algorithm was used. When taking an online test or UBT test, AI-based facial recognition technology is used to verify the tablet PC candidate's identity through a proctor terminal linked to the test server, allowing them to focus on studying online or preventing proxy testing during the test. In online learning, online testing, or UBT testing, proxy testing or visual/auditory cheating is prevented during attention-focused learning or testing by using candidate face recognition and voice recognition technology through the server.

The test paper questions in the test program (App) include subjective and/or multiple choice test questions, and for each question, not only text and images but also multimedia test questions containing text, images, VR/AR content, voice, and video are provided on the server. It is displayed on the user terminal through .

1. Artificial Intelligence Hub Platform

ㆍ Artificial intelligence hub platform based on NSD-AI (NSDevil's Artificial intelligence) platform

ㆍ Instructor grading guide through phrase recognition in language and medical education (joint research with Inje University School of Medicine and others)

ㆍ Experiential learning support system through object recognition (Fun Edu Lab, Korean Folk Village, etc.)

ㆍ Analysis of diseases in the dental education sector through tooth image analysis (joint research by Yonsei University College of Dentistry and 3 other countries)

ㆍ Based on UBT connect platform

ㆍTablet PC-based evaluation platform

ㆍ Language assessment section (speaking/listening/reading/writing)

ㆍHealth and medical evaluation sector (listening/reading/writing)

ㆍOther job evaluation categories (reading/writing)

Additionally, the learning content server 100, which includes a test management unit, registers and stores test taker information and frontal face photos from the user terminal through a wired or wireless communication network, issues QR codes to be attached to test takers to test takers, and issues QR codes for each test taker. After code recognition, a greeting is provided through TTS conversion technology, and an online test paper and a test program for conducting an online test or UBT test are provided to the candidate's terminal.

Additionally, online tests or UBT tests offer subjective tests as well as 2/3/4/5 choice multiple choice tests. The subjective test questions provide a subjective test that uses the handwriting recognition unit of the test taker's terminal equipped with a touch sensor and a display to recognize the handwriting of a stylus pen and convert it into characters to recognize the handwriting characters.

Online test papers can be entirely multiple-choice or a mix of multiple-choice/subjective.

Candidate terminals (300, 310, 311) are connected to the learning content server (100) equipped with a test management unit through a wired or wireless communication network (LAN, Wi-Fi, LTE 4G/5G), run the test program, and perform AI facial recognition/motion recognition/ Using sound recognition technology, the applicant can monitor auditory misconduct through facial recognition/motion recognition of the image of the applicant's face using the front camera (C) of the applicant's terminal, and sound recognition through the microphone and voice recognition module of the applicant's terminal. An artificial intelligence facial recognition module of a photo of the applicant's front face taken by the front camera (C) of the terminal, and a facial outline that constitutes the characteristic points of the face by recognizing facial behavior patterns taken by the front camera (C) of the candidate's terminal. It is equipped with a 5-point scale cheating prevention module for eyes 2/nose/ear 2, a voice recognition module that detects auditory cheating through voice recognition of the test taker, and is linked to a learning content server 100 equipped with a test management unit. A recording/recording program is installed on the test taker's terminal.

During an online test or UBT test, the proctor terminal receives facial photos and cheating information of a plurality of test taker terminals through the learning content server 100 equipped with a test management department, and confirms them, and the learning content server equipped with a test management department An anti-cheating alarm or message is sent to the candidate's terminal through (100).

The AI-based facial contour recognition technology used in the facial recognition module of the user terminal uses the posenet algorithm (machine learning model which allows for real-time face pose estimation).

In the embodiment, during online learning/online testing or UBT testing, a tablet PC was used as the user terminal.

The test taker terminals (300, 310, and 311) are equipped with a facial recognition module with a 5-point scale of face outline of the learner's face image and eye2/nose/ear2 facial features.

Additionally, the test taker terminal further includes a handwriting recognition unit of the test taker terminal having a display and a touch sensor that recognizes the handwriting of a stylus pen and converts it into characters for subjective test questions.

Candidate terminals (300, 310, 311) use the posenet algorithm for AI-based facial outline recognition technology used in the facial recognition module during online learning/online testing.

During an online test or UBT test, the eye2/nose/ear2 facial feature point 5-point scale facial recognition module detects the region of interest (ROI) of the face image captured in real time with the front camera of the candidate's terminal and identifies the vertex of the nose. Based on this standard size, width The outline and the facial feature points of Eye2/Nose/Ear2 are extracted, and through feature extraction and classification, the facial feature points of Eye2/Nose/Ear2 are compared to the center point (pupil) of the left/right ear and left/right eye, respectively. Calculate the Euclidean distance (d) and similarity, check whether a proxy test is performed by comparing it with a photo in the face photo database of the online test or UBT test server, and if the eye/nose 3 points are positive during the online test or UBT test. It detects head movement to the right/left depending on whether it approaches the two end ears, and detects abnormal facial behavior patterns related to cheating. If the facial outline is not recognized during face recognition, it deviates by a certain angle from the test screen. (If the distance between the eyes and ears or the distance in the corresponding direction of the nose and ears exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end), the images are taken with the camera of the test taker's terminal to prevent cheating. By recognizing facial behavior patterns, the distance between the right eye and right ear and the distance between the left eye and left ear vary depending on the angle at which the front face is turned left and right, so visual cheating is judged and visual cheating images or video data are sent to the test management department. is transmitted to the learning content server 100 having the test management unit, and after confirmation by the proctor terminal 200 linked to the learning content server 100 having the test management unit, it is sent to the learning content server 100 having the test management unit. A warning message or alarm is generated on the candidate's terminal.

(Example)

TensorFlow was used to estimate the pose of a human face on Android on a tablet PC using the Posenet algorithm model. For reference, the Posenet algorithm is a vision model that estimates a person's pose from an image or video by detecting the location of the person's face, body, arms, legs, and major body parts for action recognition. When recognizing a face, TensorFlow Lite uses the front camera of the test taker's terminal to identify the facial outline and facial feature points of eyes 2, nose, and ears 2 in real time, and creates a facial outline pose estimation model. Implemented.

The posenet algorithm used in the artificial intelligence facial recognition module uses a 5-point scale fraud prevention module for left eye (leftEye), right eye (rightEye), nose, left ear (leftEar), and right ear (rightEar). Implemented,

After finding the x,y coordinates of leftEar and the x,y coordinates of rightEar, you can find the diameter of the circle based on this and create an angel ring with ellipse(). The diameter can be easily obtained using dist() in p5js without having to use the Pythagorean theorem.

<Connecting ml5js and webcam>

<script src="https://unpkg.com/ml5@0.3.1/dist/ml5.min.js"></script>

let video;

let poseNet;

function setup() {

createCanvas(400, 400);

video = createCapture(VIDEO);

//Use to prevent the video from appearing twice

video.hide();

poseNet = ml5.poseNet(video,modelLoaded);

poseNet.on('pose',gotPoses);

console.log(ml5);

}

// Pose-related callbacks are entered

function gotPoses(poses) {

}

// To check if loading was successful

function modelLoaded() {

console.log('Model Loaded');

}

function draw() {

//Draw the webcam at position 0,0.

image(video,0,0);

}

The face area is extracted in real time from the front camera face image of the candidate's terminal, adjusted to the standard size of the frontal face photo through size correction/rotation/angle correction based on the vertex of the nose, and the standard size already stored in the test server. Compare with the learning data of the face photo DB -> Learning model of the face photo -> Extract and classify the facial outline and feature points of eyes 2/nose/ear 2, and extract and classify the face outline, eyes 2/nose/ of the face recognition data. Group clustering (cluster center k-means algorithm) of facial feature points is density estimated, and the distance (Euclidean distance) and similarity of facial feature points are calculated for online testing or face photo database on the UBT test server. Proxy testing is prevented by comparing the distance (d) and similarity of facial feature points of a standard-size frontal face photo of the applicant stored in the test to determine whether the applicant is the applicant, and visual fraud is prevented during online or UBT tests. actions are judged.

For example, if the direction of the face captured by the front camera of the test taker's terminal (tablet PC) changes or the test taker turns the face at a certain angle, the eyes 2/nose/ears 2 and the outline of the face photo are displayed. If the facial features are not recognized or the angle toward the learning content or test program changes within the effective distance (20 to 30 cm) between the user's face and the smartphone, it is recognized as failure to focus on learning or cheating during the test. The cheating detection result is transmitted to the learning content server 100 having an online test or UBT test management unit, and is transmitted to the proctor terminal 200 to be confirmed by the proctor and then sent to the learning content server 100 having a test management unit. It generates an alarm or sends a warning message to the test taker's terminal to prevent the test taker from cheating.

In addition, [Text A] data reference, the moment the warning threshold set in the predefined test taker program becomes 0, a warning message can be displayed to the test taker without a server connection and transmitted back to the proctor/cheating detection server/other test management server; Or, after the test is over, the raw data is sent to the cheating detection server/other test management server for post-processing.

In addition, the candidate terminal includes a facial recognition module, a 5-point scale for eye 2/nose/ear 2 facial features, a cheating prevention module, and the cheating prevention module recognizes the candidate's speech sounds to prevent visual cheating by the test taker. Further equipped with a voice recognition module,

During online learning, online testing, or UBT testing, when the voice recognition module hears the candidate's speech related to cheating input through the microphone of the candidate's terminal, it recognizes the voice and immediately connects the learning content server (100) equipped with a test management unit. It is transmitted and stored, and a cheating notification is sent to the supervisor terminal 200. After the supervisor confirms it, an alarm or a warning message is sent to the candidate's terminal through the learning content server to prevent cheating. Alternatively, display an alarm or warning message to the learner/candidate terminal directly from the test taker program itself without a server command, referring to the predefined distance threshold of the standard size frontal face photo of eye-to-ear distance and nose-to-ear distance; Information about cheating is transmitted to the test server and proctor terminal.

Figure 11 is a flowchart showing a tablet PC, smartphone, and PC-based attention learning method using facial contour recognition artificial intelligence in web browser-based non-face-to-face learning according to the present invention.

The present invention's tablet PC, smartphone, and PC-based attention learning method using facial contour recognition artificial intelligence in non-face-to-face learning based on a web browser is (a) registering learner information and a standard-sized frontal face photo as a learning content server; saving; (b) the learning content server issuing a QR code corresponding to user information and a standard-sized frontal face photo; (c) A user terminal equipped with an educational content viewer and a facial recognition module uses a frontal face recognition algorithm of the frontal camera image to receive the recognition result of the frontal face photo from the camera to the learning content server and correct/resize it to a standard size. Creating a standard-size frontal face photo through rotation/angle correction and comparing the learner information already registered in the server's database with the standard-size frontal face photo and facial feature points to confirm the learner's identity; (d) providing learning content from the learning content server to a user terminal of a pre-registered member; and (e) the learning content server analyzes the online learner's face image detected by the camera on each learner's user terminal, and when the gaze looking at the screen of the user terminal deviates at a certain angle or the learner's voice is heard on the learner's user terminal When recognized, it includes a step of generating a warning message or alarm to the learner's user terminal to enable focused learning,

In the method, in non-face-to-face online learning, the educational content viewer installed on the user terminal is provided with a VOD media player, a chat screen, and an online learning memo space, and the instructor's terminal uses the educational content viewer to use the learning content server ( 100) further comprising transmitting and receiving question and answer chat data to a plurality of learner terminals in a 1:N manner,

The user terminal includes a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes the facial behavior pattern of the front camera image of the user terminal and recognizes the facial outline and facial feature points on a 5-point scale of eyes 2 / nose / ears 2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end or the learner's voice is deviated from the screen by a certain angle or more. When recognized, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the user terminal to enable focused learning, and the user terminal is a voice recognition module that uses STT technology to recognize the learner's voice from a microphone. It further includes,

The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.

The learning content server 100 provides learning content during non-face-to-face online learning, regardless of time and place, and underlines or texts the online learning memo space (shared workspace) of the educational content viewer. , send/receive files with report/learning files attached, and send/receive notes to the instructor's terminal (supervisor terminal) and multiple learner terminals according to the instructor's floor control through drawings, photos, and voice recordings. provide,

During a non-face-to-face lecture, the instructor's terminal (supervisor's terminal) writes letters/graphics/memos in the shared workspace of the educational content viewer with multiple learner terminals in a 1:N manner through the learning content server 100. Write is displayed.

The learning content server 100 provides a function to recognize a QR code as a recognition code when using a smartphone.

The user terminal uses any one of a tablet PC, a smartphone, and a PC equipped with a front-facing camera, and the educational content viewer downloaded from the learning content server is installed, and the educational content viewer chats with a VOD media player. It is equipped with a screen and space for online learning notes.

The user terminal is

VOD media player that plays non-face-to-face lecture learning content;

A chat module that transmits and receives chat data between instructors and learners; and

A facial recognition module that recognizes the facial behavior pattern of the front camera image of the user terminal and recognizes the facial outline and eye2/nose/ear2 facial feature points;

If the user terminal's screen deviates by a certain angle or more (depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end, the distance between the eyes and ears or the distance in that direction exceeds a certain standard value), an alarm is triggered. It includes a learner alarm generator that generates an alarm or outputs a warning message.

The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.

The facial recognition module uses a frontal face recognition algorithm to extract facial objects from standard-size photos of the learner's face that have been size-corrected/rotated/angle-corrected during non-face-to-face online learning, and recognizes facial behavior patterns to identify the learner's face. Extract the contour and the feature points of Eye2/Nose/Ear2, and Euclidean distance (d) between the center point (pupil) of the left/right ear and left/right eye of the facial feature points of Eye2/Nose/Ear2, respectively. It calculates the similarity, compares it with the facial specific points of the user's standard-size frontal face photo in the face photo database, checks for deviation of gaze, and determines whether the 3 points of the eyes/nose are close to the 2 points of the ears at both ends. Accordingly, head movement to the right/left is detected and facial patterns related to gaze deviation are detected. When facial outlines are not recognized during face recognition, or when the user's voice deviates from the screen by a certain angle or more (eyes/nose 3 points at both ends) (Depending on whether the eye-to-ear distance or nose-to-ear distance in the corresponding direction exceeds a certain standard depending on whether the two ears are close to each other), facial behavior patterns captured by the camera of the user terminal are recognized and the angle at which the front face is turned left and right is adjusted. Since the distance between the right eye and the right ear and the distance between the left eye and the left ear vary accordingly, an alarm is generated or a warning message is output to the corresponding user terminal.

The learning content server 100

WWW server;

a control unit that controls providing online learning content to the user terminal;

A member registration unit connected to the control unit and registering member information and storing and managing ID/Passwd;

A user authentication unit connected to the control unit and authenticating using QR code/Passwd, ID/Passwd, or personal certificate;

a QR code management unit connected to the control unit and issuing and managing QR codes corresponding to user information;

a learning content provider connected to the control unit and providing the online learning content to the user terminal;

a chat server connected to the control unit and transmitting and receiving chat data between instructors and learners;

It is connected to the control unit, receives a photo of the candidate taken by the camera from the user terminal, creates a standard size frontal face photo through size correction/angle correction/rotation, and combines the user information previously stored in the server's database with the front face photo. A supervisor confirmation unit that verifies online learning qualifications by comparing face photos and verifying them on the supervisor terminal;

A face recognition unit connected to the control unit, which recognizes the outline of the learner's face and facial feature points of eyes 2/nose/ear 2 using a facial recognition module in the user terminal, and receives the face recognition result from the user terminal; and

Test information DB that stores the test papers, written answers, and scoring results of test takers' terminals; It includes a candidate DB and a face DB that store candidate information and standard-sized frontal face photos.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a chat server 111 that transmits and receives chat data between the instructor and the learner.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes a message sending/receiving service unit that transmits and receives message data to which report files of instructors and learners can be attached.

Additionally, the learning content server 100 is connected to the control unit 102 and further includes an FTP server for transmitting and receiving learning data files.

In non-face-to-face online learning, the instructor's terminal (supervisor terminal) transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server 100.

In non-face-to-face online learning, regardless of time and place, during online learning through the learning content server 100, the instructor provides learning content and underlines or writes text in the online learning memo space (shared workspace) of the educational content viewer. , send/receive files with report/learning files attached, and send/receive notes to the instructor's terminal (supervisor terminal) and multiple learner terminals according to the instructor's floor control through drawings, photos, and voice recordings. Provides,

In non-face-to-face online learning, during non-face-to-face lectures, the instructor's terminal (supervisor's terminal) shares the shared workspace of the educational content viewer with multiple learner terminals simultaneously in a 1:N manner through the learning content server 100. Text/graphic handwriting/memo writing is displayed.

Additionally, when conducting an online test or UBT test, the candidate information, the candidate's frontal face photo, and supervisor information are stored and managed in the database of the learning content server equipped with the test management department, and when the online test or UBT test is conducted, a certain test After storing the answers written on the test paper on each test taker's terminal for a period of time, at the end of the test, they are transmitted and stored from the test taker's terminal to the learning content server 100 equipped with the test management department, and the scoring results of the answers written on the test paper by the test takers are provided to the test taker's terminal. It further includes steps.

Additionally, online and UBT tests include a 2-choice/3-choice/4-choice/5-choice multiple-choice test and a subjective test that uses the handwriting recognition unit of the candidate's terminal to recognize the cursive handwriting of a stylus pen and convert it into characters to recognize handwritten characters. Includes.

In addition, the method stores the user's facial movements in the camera image of the user terminal in frame units as shown in [Text A], subtracts a defined frame, time, or number of times, and sends a warning message to the learner/candidate terminal when it reaches 0. It includes the step of displaying/exposing or sending a supervisory device or fraud detection image to the server, then resetting the number of warning triggers and starting the deduction again from the next move.

[Text A] - Facial behavior pattern information stored on the tablet PC of the test taker's terminal (e.g. facial pattern data of Seoul National University Dental School learners): Information such as date, time, angle/distance of movement, and phase (left and right) It saves every frame or hour, outputs an alarm or warning message to the user terminal based on this, confirms the supervisor and detects facial recognition fraud of the test taker - sends the relevant RAW data to the learning content server system for post-processing.

200519094106,0.02,R

200519094111,0.03,R

200519094120,0.42,R

200519094148,0.01,R

200519094150,0.21,R

200519094155,0.22,R

200519094158,0.09,R

200519094209,0.11,L

200519094214,0.09,R

200519094216,0.10,R

200519094219,0.09,R

200519094224,0.01,L

The attention learning system and method using artificial intelligence in non-face-to-face learning based on a web browser of the present invention includes an educational content viewer of a user terminal in non-face-to-face online learning having a media player and a facial outline recognition module, and a learning content server. Facial outline recognition on a 5-point scale of face contour and facial features of eyes2/nose/ears2 so that you can focus your attention on non-face-to-face lecture learning content focused on the front camera of the user terminal linked to (ubcloud server). If the gaze is diverted at a certain angle, an alarm/warning message is generated on the user's terminal using sound and messaging technology, and it provides attention learning/testing using web browser-based ubcloud artificial intelligence in non-face-to-face online learning. .

Embodiments according to the present invention are implemented in the form of program instructions that can be executed through various computer means and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include server storage, hard disks, magnetic media such as floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and floptical disks. Hardware devices configured to store and execute program instructions in storage media such as magneto-optical media, ROM, RAM, flash memory, etc. may be included. Examples of program instructions may include those produced by compilers, machine language code, as well as high-level language code that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention.

As described above, the method of the present invention is implemented as a program and can be stored on a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) in a form that can be read using computer software. ) can be stored in .

Although the present invention has been described with reference to specific embodiments of the present invention, the present invention is not limited to the same configuration and operation as the specific embodiments to illustrate the technical idea as described above, and is not limited to the technical idea and scope of the present invention. It can be implemented with various modifications, and the scope of the present invention should be determined by the claims described later.

100: Learning content server 101: WWW server
102: Control unit 103: Member register
104: User authentication unit 105: Learning content provision unit
106: QR code provision unit 107: Face recognition unit (facial outline recognition)
109: Test management department 111: Chat server
300, 310, 311: User terminal

Claims (12)

Learning content server that provides non-face-to-face online lecture learning content;
Connect to the learning content server and run the educational content viewer, and in non-face-to-face online learning, use AI facial recognition/motion recognition sound reproduction technology of the front camera image data of the user terminal to provide artificial intelligence attention-focused learning of the learning content. A user terminal with a content viewer installed; and an instructor's terminal,
The instructor's terminal transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server,
The user terminal includes a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes facial behavior patterns in the front camera image of the user terminal and recognizes facial outlines and 5-point scales of facial features of eyes2/nose/ears2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end or the learner's voice is deviated from the screen by a certain angle or more. When recognized, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the learner's user terminal to enable focused learning, and the user terminal further includes a voice recognition module that recognizes the learner's voice,
The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm.
The educational content viewer is equipped with a VOD media player, a chat screen, and an online learning memo space,
In non-face-to-face online learning, the instructor's terminal transmits and receives question and answer chat data to a plurality of learner terminals in a 1:N manner through the learning content server. An attention learning system in non-face-to-face learning based on a web browser. According to paragraph 1,
The learning content server is an attention learning system in web browser-based non-face-to-face learning that provides the function of recognizing a QR code as a recognition code when using a smartphone. According to paragraph 1,
The user terminal uses any one of a tablet PC, a smartphone, and a PC equipped with a front-facing camera, and the educational content viewer downloaded from the learning content server is installed to focus attention in web browser-based non-face-to-face learning. learning system. According to paragraph 1,
The facial recognition module uses a facial recognition algorithm to extract facial objects from a standard size corrected size/rotation/angle corrected frontal face photo of the learner during non-face-to-face online learning, and recognizes facial behavior patterns to identify the learner's face. Extract the outline and the facial feature points of Eye2/Nose/Ear2, and the Euclidean distance (pupil) of the facial feature points of Eye2/Nose/Ear2 with the center point (pupil) of the left/right ear and left/right eye, respectively ( Calculate the similarity with d), check whether the gaze is deviated by comparing it with specific points of the user's face photo in the face photo DB, and depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at both ends, the right/right Detects head movement to the left and detects facial patterns related to gaze deviation. When recognizing a face, if the facial outline is not recognized or if it deviates from the screen by more than a certain angle (the 3 points of the eyes/nose are close to the 2 points of the ears at both ends) (If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard value), facial behavior patterns captured by the camera of the user terminal are recognized, and the right eye and right ear are detected according to the angle at which the front face is turned to the left and right. An attention learning system in web browser-based non-face-to-face learning that generates an alarm or outputs a warning message to the user's terminal because the distance and the distance between the left eye and left ear are different. According to paragraph 1,
The learning content server is
WWW server;
a control unit that controls providing online learning content to the user terminal;
A member registration unit connected to the control unit and registering member information and storing and managing ID/Passwd;
A user authentication unit connected to the control unit and authenticating using QR code/Passwd, ID/Passwd, or personal certificate;
a learning content provider connected to the control unit and providing the online learning content to the user terminal;
a QR code management unit connected to the control unit and issuing and managing QR codes corresponding to user information;
a chat server connected to the control unit and transmitting and receiving chat data between instructors and learners;
It is connected to the control unit, receives a photo of the candidate taken by the camera from the user terminal, converts it into a frontal face photo through size correction/rotation/angle correction to a standard size, and compares the frontal face photo with user information already stored in the server's database. A supervisor confirmation unit where the supervisor checks online learning qualifications on the supervisor terminal;
A face recognition unit connected to the control unit, recognizes the outline of the learner's face and facial feature points of eyes2/nose/ears2 using the facial recognition module of the user terminal, and receives the face recognition result from the user terminal; and
Test information DB that stores the test papers, written answers, and scoring results of test takers'terminals; Candidate DB and face DB that store candidate information and standard-sized frontal face photos;
Attention learning system in non-face-to-face learning based on web browser including. According to clause 5,
The test program (App) and test papers are provided to the candidate's terminal and the proctor's terminal, and the candidate's information, on-site face photo, and proctor's information are managed, and the test sheet is sent to each candidate's terminal within a certain test time during an online test or UBT test. The written answers are stored and transmitted to the learning content server provided with the test management unit at the end of the test, and the web browser-based test management unit further includes a test management unit that stores and manages the test takers' written answers, scoring results, supervisor information, and test taker status information. Attention learning system in non-face-to-face learning. (a) a learning content server registering and storing learner information and a standard-sized frontal face photo from a user terminal;
(b) the learning content server issuing a QR code corresponding to user information and a frontal face photo;
(c) A user terminal equipped with an educational content viewer and a facial recognition module uses the front-facing face recognition algorithm of the front-facing camera image to receive the recognition result of the front-facing face photo from the camera to the learning content server, along with pre-registered learner information and standards. Comparing the size of the frontal face photo and the facial features to confirm the learner's identity;
(d) providing learning content from the learning content server to a user terminal of a pre-registered member; and
(e) The learning content server analyzes the online learner's face image detected by the camera on each learner's user terminal to enable focused learning when the gaze is off at a certain angle or when the learner's voice is recognized by the learner's user terminal. Includes a step of generating a warning message or alarm to the learner's user terminal,
In non-face-to-face online learning, the educational content viewer installed on the user terminal is equipped with a VOD media player, a chat screen, and an online learning memo space, and the instructor's terminal is connected to a plurality of learner terminals in a 1:N manner through the learning content server. It further includes the step of transmitting and receiving question and answer chat data,
The user terminal includes a VOD media player that plays non-face-to-face lecture learning content; A chat module that transmits and receives chat data between instructors and learners; A facial recognition module that recognizes the facial behavior pattern of the front camera image of the user terminal and recognizes the facial outline and facial feature points on a 5-point scale of eyes 2 / nose / ears 2; and If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at either end or the learner's voice is deviated from the screen by a certain angle or more. When recognized, it includes a learner's alarm generator that generates an alarm or outputs a warning message to the user terminal to enable attention-focused learning, and the user terminal further includes a voice recognition module that recognizes the learner's voice,
The facial recognition module is an AI-based facial outline recognition technology that uses the posenet algorithm and is an attention learning method in web browser-based non-face-to-face learning. In clause 7,
An attention learning method in web browser-based non-face-to-face learning, further comprising providing the learning content server with a function to recognize a QR code as a recognition code when using a smartphone. In clause 7,
The user terminal uses any one of a tablet PC, a smartphone, and a PC equipped with a front-facing camera, and the educational content viewer downloaded from the learning content server is installed to focus attention in web browser-based non-face-to-face learning. How to learn. In clause 7,
The facial recognition module uses a frontal face recognition algorithm to extract facial objects from standard-size photos of the learner's face that have been size-corrected/rotated/angle-corrected during non-face-to-face online learning, and recognizes facial behavior patterns to identify the learner's face. Extract the outline and the facial feature points of Eye2/Nose/Ear2, and the Euclidean distance (pupil) of the facial feature points of Eye2/Nose/Ear2 with the center point (pupil) of the left/right ear and left/right eye, respectively ( Calculate the similarity with d), check whether the gaze is deviated by comparing it with specific points of the user's frontal face photo in the face photo DB, and depending on whether the 3 points of the eyes/nose are close to the 2 points of the ears at both ends, the right / Detects head movement to the left and detects face patterns related to gaze deviation. If the facial outline is not recognized when recognizing a face, or if it deviates from the screen by more than a certain angle (the 3 points of the eyes/nose are close to the 2 points of the ears at both ends) (If the distance in the corresponding direction of the eye-to-ear distance or nose-to-ear distance exceeds a certain standard value) By recognizing the facial behavior pattern captured by the camera of the user terminal, the right eye and right eye are detected according to the angle at which the front face is turned to the left and right. An attention learning method in web browser-based non-face-to-face learning that generates an alarm or outputs a warning message on the user terminal because the distance between the ears and the left eye and left ear are different. In clause 7,
The learning content server is
WWW server;
a control unit that controls providing online learning content to the user terminal;
A member registration unit connected to the control unit and registering member information and storing and managing ID/Passwd;
A user authentication unit connected to the control unit and authenticating using QR code/Passwd, ID/Passwd, or personal certificate;
a learning content provider connected to the control unit and providing the online learning content to the user terminal;
a QR code management unit connected to the control unit and issuing and managing QR codes corresponding to user information;
It is connected to the control unit, receives a photo of the candidate taken by the camera from the user terminal, converts it into a frontal face photo through size correction/rotation/angle correction to a standard size, and compares the frontal face photo with user information already stored in the server's database. A supervisor confirmation unit where the supervisor checks online learning qualifications on the supervisor terminal;
A face recognition unit connected to the control unit and recognizing the outline of the learner's face and eye2/nose/ear2 feature points using a facial recognition module in the user terminal and receiving the face recognition result from the user terminal; and
Test information DB that stores the test papers, written answers, and scoring results of test takers'terminals; Candidate DB and face DB that store candidate information and standard-sized frontal face photos;
Attention learning method in web browser-based non-face-to-face learning including. In clause 7,
The learning content server stores and manages the candidate information, the candidate's on-site face photo, and the supervisor information in the database of the test server, and during the online test or UBT test, the test paper and answers are stored on each test taker's terminal for a certain test time. Attention is focused in web browser-based non-face-to-face learning, which further includes the step of transmitting and storing the learning content from the test taker's terminal to the learning content server provided with the test management unit at the end of the test, and providing the scoring results of the answers written by the test takers to the test taker's terminal. How to learn.