[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN108073888A - A kind of teaching auxiliary and the teaching auxiliary system using this method - Google Patents

A kind of teaching auxiliary and the teaching auxiliary system using this method Download PDF

Info

Publication number
CN108073888A
CN108073888A CN201710667590.7A CN201710667590A CN108073888A CN 108073888 A CN108073888 A CN 108073888A CN 201710667590 A CN201710667590 A CN 201710667590A CN 108073888 A CN108073888 A CN 108073888A
Authority
CN
China
Prior art keywords
student
classroom
module
teaching auxiliary
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201710667590.7A
Other languages
Chinese (zh)
Inventor
王书强
王永灿
王兆哲
胡勇
杨岳
胡明辉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Sibiku Technology Co ltd
Shenzhen Institute of Advanced Technology of CAS
Original Assignee
Shenzhen Sibiku Technology Co ltd
Shenzhen Institute of Advanced Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Sibiku Technology Co ltd, Shenzhen Institute of Advanced Technology of CAS filed Critical Shenzhen Sibiku Technology Co ltd
Priority to CN201710667590.7A priority Critical patent/CN108073888A/en
Publication of CN108073888A publication Critical patent/CN108073888A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06QINFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/10Services
    • G06Q50/20Education
    • G06Q50/205Education administration or guidance
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/174Facial expression recognition

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Strategic Management (AREA)
  • Educational Technology (AREA)
  • Tourism & Hospitality (AREA)
  • Data Mining & Analysis (AREA)
  • Educational Administration (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Evolutionary Biology (AREA)
  • Economics (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Computation (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Human Resources & Organizations (AREA)
  • Marketing (AREA)
  • Primary Health Care (AREA)
  • General Business, Economics & Management (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Image Analysis (AREA)

Abstract

A kind of teaching auxiliary system the invention discloses teaching auxiliary and using this method, wherein teaching auxiliary can provide the requirement of higher image recognition precision and reduction algorithm to hardware by using trained depth tensor row network model to carry out behavioral value to the student in the classroom image, and can be used on embedded device, reduce the use cost of teaching auxiliary;Also had the advantages that using the teaching auxiliary system of the teaching auxiliary in the present invention simultaneously similary.

Description

A kind of teaching auxiliary and the teaching auxiliary system using this method
Technical field
A kind of tutor auxiliary platform the present invention relates to tutor auxiliary platform field more particularly to teaching auxiliary and using this method System.
Background technology
In general education activities, due to the student to attend class and the quantitative proportion great disparity of teacher, teacher exists There is no too many time and efforts when giving lessons by observing attending class behavior and expression and judging the study shape of student for each student State.This allows for teacher and can not accurately not understand attending class for each student state and being received journey to this professor's content Degree.It easilying lead to teacher on classroom and says teacher, student chats student, and then entire education activities is allowed to be torn and are come, So that teacher with a definite target in view can not impart knowledge to students, it is serious to affect quality of instruction and efficiency.Thus it is possible in student The teaching auxiliary system that upper class hour uses is always educational circles' Important Problems of interest.Designed teaching auxiliary system auxiliary is given lessons Teacher smoothly carries out education activities.Functionality is emphasized in current teaching auxiliary system research, mainly autonomous from being provided for student Academic environment provides sufficient education resource for student, mitigates the several aspect expansion of workload of teacher.With different technologies Means design intelligent auxiliary system to improve give lessons effect and the learning efficiency of student of teacher.
In the prior art, the Chinese invention patent of Publication No. CN106097790A discloses a kind of tutor auxiliary platform dress Put, by image recognition technology identify education activities in image, and then come judge student attend class whether do it is unrelated with attending class Thing, and according to recognition result teacher is notified to do respective handling.
Since the prior art does not disclose the method and process of its picture recognition module identification image, also without disclosing it How picture recognition module is realized, and conventional images are compared with pre-stored image, and judge comparison result.Technical staff according to The prior art scheme can not be implemented as the technique effect that teaching process is aided in.Therefore, existing tutor auxiliary platform side Method Shortcomings.
The content of the invention
In order to solve the above technical problems existing in the prior art, have it is an object of the invention to provide one kind higher Image recognition precision teaching auxiliary and using this method teaching auxiliary system.
In order to solve the above-mentioned technical problem, the technical solution adopted in the present invention is:A kind of teaching auxiliary, including with Lower sequential steps:
S1. the classroom image of the real-time collection site of acquisition module, and it is transferred to identification module;
S2. the identification module analyzes the classroom image, and judges abnormal behavior in the classroom image Student;
S3. the recognition result of the identification module is notified teacher by reminding module;
Comprise the following steps in the step s2:
S21. the identification module using trained depth tensor row network model come in the classroom image It is raw to carry out behavioral value.
Preferably, the step s2 is further comprising the steps of:
S22. the identification module using trained depth tensor row network model come in the classroom image It is raw to carry out Expression Recognition.
Preferably, the step s22 specifically includes following steps:
S221. each is identified from the classroom image that the acquisition module collects by Face datection subelement Raw human face region;
S222. Expression Recognition is done to the human face region detected by convolutional neural networks grader.
Preferably, comprise the following steps in step s1:
S11. image collecting device is installed in left, center, right region of the acquisition module in front of classroom respectively;
S12. described image acquisition module is using all student's upper part of the body images in class as acquisition target.
Preferably, it is further comprising the steps of:S4. memory module synchronously achieves the recognition result.
Preferably, comprise the following steps in the step s4:
S41. the corresponding recognition result of each student is formulated to student's electronic record by class;
S42. draw student according to student's electronic record to attend class condition curve, be combined at that time in order to teacher The content and total marks of the examination of professor targetedly teaches student.
Preferably, it is further comprising the steps of before step s1:
Q1. data set is built;
Q2. the depth tensor row network model is trained.
Preferably, the step q1 comprises the following steps:
Q11. the acquisition module shoots the classroom image for a long time in classroom and stores;
Q12. the student pictures that there is exception are chosen to be labeled.
Preferably, the step q2 comprises the following steps:
Q21. the off-note in the student pictures marked is extracted by the multilayer convolutional layer of neural network model, The off-note obtains output predicted value with the full articulamentum weight matrix computing after decomposing;
Q22. the error of the output predicted value and the true mark value of abnormal behaviour student in the student pictures is formed Loss function;
Q23. network parameter is adjusted according to the loss function, obtains trained depth tensor row network model.
In order to solve the above-mentioned technical problem, the present invention also provides a kind of teaching auxiliary system, it is provided with:Acquisition module, with The identification module of the acquisition module connection, the reminding module being connected with the identification module;
The acquisition module, for real-time collection site classroom image and be transferred to identification module;
The identification module for analyzing the classroom image, and judges abnormal behavior in the classroom image Student;The identification module includes:
Behavioral value unit, for using trained depth tensor row network model come in the classroom image It is raw to carry out behavioral value;
The reminding module, for the recognition result of the identification module to be notified teacher.
Preferably, it is additionally provided with:The memory module being connected with the identification module;The memory module, for synchronously depositing The shelves recognition result is gone forward side by side edlin analysis;
The identification module further includes:
Expression Recognition unit, for using trained depth tensor row network model come in the classroom image It is raw to carry out Expression Recognition;
The Expression Recognition unit includes Face datection subelement and convolutional neural networks grader.
Compared with prior art, teaching auxiliary of the invention, by using trained depth tensor row network mould Type can provide higher image recognition precision and reduction algorithm pair to carry out behavioral value to the student in the classroom image The requirement of hardware, and can be used on embedded device, reduce the use cost of teaching auxiliary.
Further, the present invention also using trained depth tensor row network model come in the classroom image It is raw to carry out Expression Recognition so that teaching auxiliary system is to the abnormal behaviour accuracy of identification higher of student's upper class hour.
Using the teaching auxiliary system of this method, similarly have the advantages that above-mentioned.
Description of the drawings
Fig. 1 is a kind of basic flow chart of teaching auxiliary;
Fig. 2 is a kind of detail flowchart of teaching auxiliary;
Fig. 3 is the teaching auxiliary system configuration diagram using Fig. 1 teaching auxiliaries;
Fig. 4 is the complete configuration diagram of Fig. 3 teaching auxiliary systems;
Fig. 5 folds and is fused to three rank tensor schematic diagrames for full link weight matrix;
Fig. 6 carries out tensor row decomposition diagram for three rank tensors;
Fig. 7 is tensor row decomposition diagram;
Fig. 8 is the tensor row decomposition diagram of matrix;
Fig. 9 lays schematic diagram for acquisition module;
Figure 10 is depth tensor row network architecture schematic diagram used by behavioral value;
Figure 11 be Expression Recognition used by depth tensor row network architecture adopt schematic diagram.
Specific embodiment
Below with reference to attached drawing 1 to attached drawing 11, various embodiments of the present invention are further described in detail.
A kind of teaching auxiliary as shown in Figure 1, including following sequential steps:
S1. the classroom image of the real-time collection site of acquisition module, and it is transferred to identification module.
S2. the identification module analyzes the classroom image, and judges abnormal behavior in the classroom image Student.
S3. the recognition result of the identification module is notified teacher by reminding module.
Comprise the following steps in the step s2:
S21. the identification module using trained depth tensor row network model come in the classroom image It is raw to carry out behavioral value.
Specifically, the depth tensor row network model used in step s21 to traditional full connection layer matrix by doing tensor Row are decomposed and got, and the parameter amount of the full connection layer matrix tensor of huge compression improves efficiency of algorithm, reduces algorithm to hardware It is more convenient to use simple and cost can be reduced it is required that system is facilitated to be disposed in the form of embedded device, it is auxiliary beneficial to this teaching The large-scale promotion of auxiliary system.
As shown in Figure 2, the step s2 is further comprising the steps of:
S22. the identification module using trained depth tensor row network model come in the classroom image It is raw to carry out Expression Recognition.
Step s22 has reseted image and has identified this core algorithm, and behavioral value and table are carried out to student pictures by combining Feelings identify, depth tensor row network model is made to obtain better accuracy of identification and efficiency.It is reduced in depth tensor row network model Model parameter amount improves and effectively raises this teaching auxiliary system on the basis of the robustness of system and detect classroom in real time The speed of upper student's abnormal behaviour and expression.
In the present embodiment, the step s22 specifically includes following steps:
S221. each is identified from the classroom image that the acquisition module collects by Face datection subelement Raw human face region.
S222. Expression Recognition is done to the human face region detected by convolutional neural networks grader.
In concrete operations, since human face expression feature is relatively thin, identification module is inconvenient directly to extract expressive features, Therefore, the present invention realizes the identification of expression by step s221 and step s222.First by Face datection subelement from figure As detecting each student's human face region in classroom picture that acquisition module collects, then by convolutional neural networks grader to inspection Each face area image block measured does Expression Recognition.
As shown in Figure 9, comprise the following steps in step s1:
S11. image collecting device is installed in left, center, right region of the acquisition module in front of classroom respectively.
In other embodiments, left and right two regions or multiple regions installation diagram picture in front of classroom can also be used Harvester, to prevent single direction shooting from easily thering is student to be blocked.
Meanwhile in a preferred embodiment, under state of normally giving lessons Most students be not in substantially it is puzzled, stupefied, The abnormal behaviours such as bored, therefore can be the time interval that each image collecting device sets shooting, to reduce the sample rate of image, Save corresponding processing and storage resource.
S12. described image acquisition module is using all student's upper part of the body images in class as acquisition target.
In the specific implementation, student at school when behavior and expressive features can be extracted with upper part of the body image substantially and It is identified, to be image-region that target can targetedly shoot that feature compares enrichment above the waist.
In the present embodiment, it is further comprising the steps of:S4. memory module synchronously achieves the recognition result.
In the present embodiment, by synchronizing storage to recognition result, further identification can be tied from general direction Fruit carries out analysis and utilization.Such as:It is analyzed according to recognition result and assesses teaching efficiency and analyze the learning curve of student, Ke Yigeng Targetedly carry out education activities, next teaching can more be shot the arrow at the target, it is whole to improve teaching level and quality.
Preferably, comprise the following steps in the step s4:
S41. the corresponding recognition result of each student is formulated to student's electronic record by class.
Contribute to attend class to each student the detection recognition result progress statistical analysis of state and existing to student for active The state of listening to the teacher in school avoids only listening to the teacher this passive mode hysteresis of state to judge student by the achievement of student into line trace The drawbacks of.
S42. draw student according to student's electronic record to attend class condition curve, be combined at that time in order to teacher The content and total marks of the examination of professor targetedly teaches student.
It is also possible to student's electronic record is combined with the teaching evaluation of teacher, improve at present with student examination into Achievement for Classroom Teaching Quality Assessment referring especially to it is excessively unilateral the drawbacks of.
In the present embodiment, it is further comprising the steps of before step s1:
Q1. data set is built.
In specific implementation, can be divided into for behavioral value build data set and be Expression Recognition build data set.
Specifically, when building data set for behavioral value, can one suitable data set of structure be correctly to detect The basis of abnormal behaviour student is directly related to the height of system identification performance.We are long in multiple classrooms using acquisition module Time shooting classroom is attended class situation, then therefrom chooses that there are the pictures of abnormal behaviour student to be labeled, wherein abnormal behaviour Refer to all behaviors for showing as not listening to the teacher conscientiously, such as sleep, speak, do little trick, is stupefied.Since there may be occlusion issues And the limitation of single-view, we use the image acquisition device picture at the visual angle of left, center, right three, mark respectively.And Simple process is done to picture, the fixed dimension of input is used to be adjusted to fit model to facilitate in training network model.
In other embodiments, the expressive features such as absorbed, in high spirits, thinking can also be extracted, and to depth Tensor row network model is trained so that the behavior conscientiously listened to the teacher can also be come out by the Model Identification.
When building data set for Expression Recognition, since we do expression recognition and are carried out in two steps herein, do first Face datection, then Expression Recognition is done, we need to build two datasets, and one is classroom Face datection data set, another is Classroom student attends class expression data collection.
Face datection data set
Can accurately accurately to detect student face in real time from the classroom image that image capture module collects, I Build a small-sized classroom Face datection data set.We shoot classroom for a long time using image capture module in multiple classrooms The classroom image that the situation of attending class collects, is labeled picture, provides face location in picture, and simple place is done to picture Reason, is adjusted to fit the fixed dimension of mode input, is used with facilitating in training network model.
Student attends class expression data collection
The more accurate state of listening to the teacher for understanding each student on classroom in real time of the teacher that attends class for convenience, meets student and attends class Expression Recognition demand, we listen to the teacher this scene for student classroom, and one student of structure attends class expression data collection.From what is collected In the image of classroom, student's facial expression picture block is intercepted out, provides corresponding conscientious degree correlation expression label of listening to the teacher, be such as absorbed in, In high spirits, thinking, puzzled, stupefied, bored etc..Facilitate listening to the teacher for the more convenient careful every student of grasp of teacher State and course grasp situation and attitude, make processing in real time and adjustment.
Q2. the depth tensor row network model is trained.
In specific implementation, the training of behavioral value and Expression Recognition can be separated and carried out.It is differed only in using not Same data set is trained.
Specifically, being based on " depth tensor row network " classroom student's abnormal behaviour identification neural network model.It is logical first It crosses multilayer convolutional layer to learn to extract the behavioural characteristic of classroom picture middle school student automatically, in the classroom behavior feature letter that study is used to arrive It ceases the full articulamentum after TT decomposes (tensor row decompose) student classroom behavior is identified, is tested with abnormal classroom behavior Student.
As shown in Figure 7, tensor row decompose (tensor train decomposition, TT-decomposition) It is a kind of tensor resolution model, by the product representation of several matrixes of each element of tensor.Assuming that there are d rank tensors(IkRepresent the dimension of kth rank), tensorTensor row be decomposed into:
WhereinIt is tensorThe corresponding nuclear matrix of kth class, scale rk-1′rk, k=1,2 ... d, r0= rd=1;(r0, r1... rd) it is d rank tensorsCorresponding TT-rank during tensor row is carried out, actuallyIt is that scale is rk-1 Ik rkThree rank tensors, soIt is called core tensor.
As shown in Figure 8, the tensor row of matrix decompose, it is assumed that matrix A ∈ RM×N, select reconfiguration scheme, such as reconfiguration scheme:After selected reconfiguration scheme, the tensor row of matrix decompose is mapped to d rank tensors by matrix firstAgain to tensorIt carries out tensor row to decompose, i.e.,
As shown in Figure 2, the step q1 comprises the following steps:
Q11. the acquisition module shoots the classroom image for a long time in classroom and stores.
Q12. the student pictures that there is exception are chosen to be labeled.
May be relatively fewer due to containing abnormal behaviour image data, to avoid model over-fitting, and enhance model to light According to the antijamming capability of the factors such as variation, we do data enhancing to classroom student's abnormal behaviour image data of acquisition mark. The processing such as contrast, RGB channel intensity, plus noise are changed to picture respectively, increase image data sample size and species.
In the present embodiment, the step q2 comprises the following steps:
Q21. the off-note in the student pictures marked is extracted by the multilayer convolutional layer of neural network model, Off-note obtains output predicted value with the full articulamentum weight matrix computing after decomposing.
The model structure (only illustrating herein by taking 3 layers of convolution as an example) of depth tensor row network as shown in Figure 10;Structure The step of depth tensor row network model is:
1. initialize network model parameter.
2. the picture in classroom student's abnormal behaviour data set of structure is input to the model to be trained.
3. picture passes through continuous convolution pond, the tensor A of S x S x m is exported in last layer of convolutional layer, i.e., artwork Piece has marked off the grid of a S x S, and each grid cell corresponds to a part for former classroom picture, the figure in each grid Piece feature corresponds to a m dimensional vector in the tensor.
4. passing through improved full link, the tensor of a S x S x (5a), i.e., the corresponding a of each grid cell are exported A abnormal behaviour student detects bounding box coordinates (x, y, w, h) with being detected as the confidence level of abnormal behaviour student in identification frame.Its Middle x and y identifies frame center point coordinate for abnormal behaviour student, and w and h are respectively that abnormal behaviour student identifies the wide and high of frame, and Coordinate is normalized, makes it between 0 to 1.
It is wherein improved to be connected as doing tensor row (TT) decomposition to traditional full connection layer matrix entirely, so as to which huge compression is complete Layer parameter amount is connected, efficiency of algorithm is improved, reduces the requirement to hardware so that can be used on embedded device.For this teaching Auxiliary system improves the speed of detection classroom abnormal behaviour student in real time, and system is facilitated to be disposed in the form of embedded device, It is more convenient simple and cost can be reduced, identify that the extensive of teaching auxiliary system pushing away beneficial to this classroom abnormal behaviour student Extensively.
It is a kind of tensor resolution model that tensor row, which decompose (TT decomposition), by each element of tensor with several matrixes Product representation.The tensor row of matrix, which decompose, need to first select reconfiguration scheme, and matrix is mapped to d rank tensors first, then to tensor Tensor row are carried out to decompose.It is to do tensor row to full articulamentum weight matrix to decompose herein, is below the detailed solution to the process It releases (for convenience of explanation, we substitute into the citing of some parameters, but specific implementation is not limited to design parameter)
In the present embodiment, full articulamentum weight matrix tensor row decomposition step is:
1. the row and column of full connection weight matrix as shown in Figure 5, is folded into d virtual dimensions;It is assumed herein that The classroom image that S=4, m=50, n=49 in network model, i.e. image capture module are collected extracts behind successively convolution pond To 4x4x50=800 feature, next hidden layer has 4x4x49=784 hidden node, then the full articulamentum weight parameter is The matrix of 800x784.For convenience of expression, d=3 is taken, the row and column of full connection weight matrix is folded into 3 virtual dimensions Degree, as shown in drawings.
2. as shown in Figure 5, the corresponding virtual dimension of ranks is merged, will the remodeling of connection weight matrix be entirely D rank tensors;3 rank tensors for 700x32x28 are remolded by the weight matrix of examples detailed above method then original 800x784.
3. as shown in Figure 6, define tensor the column rank r, wherein r of the d ranks tensorkRepresent original tensor remove before (k- 1) rank of matrix being unfolded after rank effect along tensor kth rank, wherein r0=rd=1 is constraints;Tensor row defined herein Order is 3.
The tensor row exploded representation of full articulamentum weight matrix is obtained 4. the d ranks tensor is carried out tensor row and is decomposed, i.e.,WhereinIt is that scale is rk-1 Ik rkThree rank tensors, IkRepresent the dimension of high order tensor kth rank.At this In example, i.e. the 3 rank tensors of original 700x32x28 are decomposed for 3 core tensors of 1x700x3,3x32x3,3x28x1.Quan Lian It connects layer weight and has decreased to 2472 parameters by 627200 original parameters.
More intuitively to represent that TT decomposes (tensor row decompose) to the compression effectiveness of full articulamentum weight parameter amount, now will Parameter scale calculates such as following table before and after tensor row decompose under several remodeling schemes.It can be seen that by result of calculation in table, full articulamentum Weight parameter parameter amount after tensor row decompose has dropped hundreds and thousands of times, can improve efficiency of algorithm, reduce the requirement to hardware, Facilitate realization of this teaching auxiliary system on embedded device, improve the real-time of student's abnormal behaviour on detection classroom.
Q22. the damage of predicted value and the error composition of the true mark value of abnormal behaviour student in the student pictures is exported Lose function;
Q23. network parameter is adjusted according to loss function, obtains trained depth tensor row network model.
5. with back-propagation algorithm, between the true mark value of abnormal behaviour student in the predicted value and artwork of output The loss function L (loss function uses error of sum square loss function herein, hereinafter specific to introduce) that error is formed, adjustment Network parameter, until designated precision.Then network parameter is preserved.
For loss function using error of sum square loss function including 3 parts, coordinate anticipation function includes abnormal behaviour The confidence level anticipation function of the confidence level anticipation function of the identification frame of student and the identification frame not comprising abnormal behaviour student.
Wherein, x, y are the center position coordinates that abnormal behaviour student identifies frame, and w, h are that abnormal behaviour student identifies frame It is wide and high,To judge whether j-th in i-th of grid identification frame is responsible for detection,To determine whether abnormal row It is fallen within for students' center in grid i, lcoordWeight, l are predicted for coordinatenoobjNot include the identification frame of abnormal behaviour student Confidence weight.
In a preferred embodiment, as shown in attached drawing 10, attached drawing 11, depth tensor row network mould when being trained for Expression Recognition During type, Face datection network model, the classroom unusual checking model of Face datection and behavioral value submodule are trained first It is similar, wherein it will change Face datection data set by abnormal behaviour data set in classroom, it is defeated according to the picture of concentration using face testing number Enter model training, repeat the training process of 1-5 in above-mentioned behavioral value submodule so that model can learn face spy automatically Sign, detects student's face location automatically from the image of classroom.
Secondly convolutional neural networks (CNN) grader used during training classroom face human facial expression recognition.By foregoing structure The student built expression data of attending class concentrates the student face picture block input Expression Recognition grader with expression label, and expression is known Other network model is trained.Expression Recognition network model such as attached drawing 11.
1. initialize Expression Recognition network model parameter.
2. by the student of structure attend class expression data concentrate the student face picture block with expression label be input to the model It is trained.
3. student face picture block passes through continuous convolution pond, facial expression feature is extracted.
4. passing through improved full link, the student face picture block expression label of prediction is exported.Herein also to connecting entirely Layer weight matrix does TT decomposition.Detailed process is discussed in detail in behavior detection sub-module in (4), and details are not described herein again.
5. with back-propagation algorithm, according to the loss that error is formed between the predicted value of output and true mark expression label Function L adjusts network parameter, until designated precision, then preserves network parameter.
In other embodiments, the step of being also the accuracy of retrieval model study, further include model measurement.
When being tested for behavioral value, above-mentioned trained network model parameter is imported into behavior in identification module and is examined The depth tensor row network of submodule is surveyed, inputs the classroom picture gathered in real time by image capture module, is detected in real time in picture Whether abnormal behaviour student is had, if there is then marking and recognition result being notified teacher by reminding module, and by storage mould Block achieves, so that subsequently data are further analyzed with excavation.Whether it is abnormal row that abnormal behaviour is provided according to network model Determined for whether probability is more than given probability threshold value, default probability threshold value by repeatedly test provide one it is rational meet it is big Many preferable balance sensitivities of energy and the value of accuracy, teacher can subsequently make the appropriate adjustments according to personal considerations, so that this religion It is humanized to learn auxiliary system.During test can according to there are the problem of, make the appropriate adjustments in detail, to reach system To optimum state, then input actual use.
When being tested for Expression Recognition, above-mentioned trained network model parameter is imported into expression in identification module and is known Small pin for the case module inputs the classroom picture gathered in real time by image capture module, detects figure by Face datection network model first All face locations in piece, then fixed size input expression after the face picture block simple process detected, will be adjusted to and known Other network model identification student attends class expression.So that model detects face and identifies its expressive features automatically, so that model can be thrown Enter actual use, test and analyze the expression information of student on classroom in real time, bonding behavior detection module is as a result, the teacher that conveniently attends class The more accurate state of attending class for understanding each student on classroom in real time, teacher can more shoot the arrow at the target, improves teaching matter Amount and efficiency.
In order to solve the above-mentioned technical problem, the present invention also provides a kind of teaching auxiliary system, it is provided with:Acquisition module, with The identification module of the acquisition module connection, the reminding module being connected with the identification module.
The acquisition module, for real-time collection site classroom image and be transferred to identification module.
Acquisition module, as shown in Figure 9, image capture module acquisition target is all student's upper part of the body pictures of class.It adopts Mode set be by front of classroom the left, center, right top of wall image collecting device is installed respectively, adjust shooting angle, To prevent from blocking and integrate multiple visual angles, the time interval that image collecting device is set to shoot every time, at the picture collected Identification module is transferred to after managing into size needed for identification module, data are provided to carry out classroom behavior identification;
Identification module for analyzing the classroom image, and judges abnormal behavior in the classroom image It is raw;It is specifically included with lower unit:
Behavioral value unit, for using trained depth tensor row network model come in the classroom image It is raw to carry out behavioral value.
Specifically, the purpose of identification module is to identify specific classroom behavior and the expression of the student that attends class in image capture module Judge whether student is conscientiously listening to the teacher, understand acceptance level of the student to lecture contents.Behavioral value method is first acquisition class It attends class student pictures data in hall, artificial mark is done to picture, marks wherein exceptional student, i.e., do not listen to the teacher conscientiously student, specifically Including sleeping, speaking, doing little trick, stupefied etc..Reuse the classroom behavior image data collection training depth tensor row net of structure Network model so that identification module can learn picture feature automatically, detect student's abnormal behaviour in picture.It will finally train Model input actual use, 3 images coming are transmitted in real-time image acquisition acquisition module, and (this patent is by taking three width images as an example Illustrate, under conditions of hardware device license, multiple image can be gathered in real time), the student detected respectively in picture is abnormal Behavior, and abnormal behavior student is outlined according to given probability threshold value.
The reminding module, for the recognition result of the identification module to be notified teacher.
Reminding module, recognition result synthesis is notified teacher by reminding module in some way in real time, if 3 angles Image it is all without exception, do not notify, teacher can adjust identification sensitivity by adjusting probability threshold value.Teacher is carried receiving The acceptance level of listen to the teacher state and the content taught it of classroom student can be understood after showing in real time, it can be based on this It is not that corresponding countermeasure is putd question to or taken to good classmate's emphasis to wherein acceptance level.
Preferably, it is additionally provided with:The memory module being connected with the identification module;The memory module, for synchronously depositing The shelves recognition result is gone forward side by side edlin classification.
Memory module, memory module be the final result by all identifications of the system using class as mesh, student is class, with The form of personal files of students is stored, and school can make full use of these electronic records, therefrom excavates useful information, and one Aspect can integrally receive situation to analyze and assess the deficiency in teaching according to student, on the other hand can analyze student Curve is practised, the bad true cause of student performance is found, can targetedly carry out looking into scarce mending-leakage.
The identification module further includes:
Expression Recognition unit, for using trained depth tensor row network model come in the classroom image It is raw to carry out Expression Recognition.
The Expression Recognition unit includes Face datection subelement and convolutional neural networks grader.
Expression Recognition submodule block method is similar with behavioral value, the difference is that the abnormal expression of mark carries out algorithm instruction Practice.It is described in this patent by two submodules in a manner that identical algorithms model separates parallelism recognition, but change can also be passed through Two tasks by multitask loss function are fused in same model and identified, are not specifically described herein by loss function, but Also within the scope of this patent.
Present invention scheme claimed solves old by being analyzed and processed to aid in classroom image well The technical issues of Shi Jinhang education activities, avoiding existing teaching equipment and excessively relying on external pattern recognition device causes firmly Part requirement is high and identifies the defects of inaccurate, improves the efficiency of teacher's teaching.
The above is only presently preferred embodiments of the present invention, is not intended to limit embodiment of the present invention, this field is general Logical technical staff's central scope according to the present invention and spirit can very easily carry out corresponding flexible or modification, therefore originally The protection domain of invention should be subject to the protection domain required by claims.

Claims (11)

1. a kind of teaching auxiliary, including following sequential steps:
S1. the classroom image of the real-time collection site of acquisition module, and it is transferred to identification module;
S2. the identification module analyzes the classroom image, and judges the student of abnormal behavior in the classroom image;
S3. the recognition result of the identification module is notified teacher by reminding module;
It is characterized in that, comprise the following steps in the step s2:
S21. the identification module using trained depth tensor row network model come to the student in the classroom image into Row behavioral value.
2. a kind of teaching auxiliary as described in claim 1, which is characterized in that the step s2 is further comprising the steps of:
S22. the identification module using trained depth tensor row network model come to the student in the classroom image into Row Expression Recognition.
3. a kind of teaching auxiliary as claimed in claim 2, which is characterized in that the step s22 specifically includes following step Suddenly:
S221. identify each student's from the classroom image that the acquisition module collects by Face datection subelement Human face region;
S222. Expression Recognition is done to the human face region detected by convolutional neural networks grader.
4. a kind of teaching auxiliary as described in claim 1, which is characterized in that comprise the following steps in step s1:
S11. image collecting device is installed in left, center, right region of the acquisition module in front of classroom respectively;
S12. described image acquisition module is using all student's upper part of the body images in class as acquisition target.
5. a kind of teaching auxiliary as described in claim 1, which is characterized in that further comprising the steps of:S4. memory module It is synchronous to achieve the recognition result.
6. a kind of teaching auxiliary as claimed in claim 5, which is characterized in that comprise the following steps in the step s4:
S41. the corresponding recognition result of each student is formulated to student's electronic record by class;
S42. draw student according to student's electronic record to attend class condition curve, be taught at that time in order to which teacher combines Content and total marks of the examination student is targetedly taught.
7. a kind of teaching auxiliary as described in claim 1, which is characterized in that further comprising the steps of before step s1:
Q1. data set is built;
Q2. the depth tensor row network model is trained.
8. a kind of teaching auxiliary as claimed in claim 7, which is characterized in that the step q1 comprises the following steps:
Q11. the acquisition module shoots the classroom image for a long time in classroom and stores;
Q12. the student pictures that there is exception are chosen to be labeled.
9. a kind of teaching auxiliary as claimed in claim 8, which is characterized in that the step q2 comprises the following steps:
Q21. the off-note in the student pictures marked is extracted by the multilayer convolutional layer of neural network model, it is described Off-note obtains output predicted value with the full articulamentum weight matrix computing after decomposing;
Q22. the damage that the error of the output predicted value and the true mark value of abnormal behaviour student in the student pictures is formed Lose function;
Q23. network parameter is adjusted according to the loss function, obtains trained depth tensor row network model.
10. a kind of teaching auxiliary system, which is characterized in that be provided with:Acquisition module, the identification mould being connected with the acquisition module Block, the reminding module being connected with the identification module;
The acquisition module, for real-time collection site classroom image and be transferred to identification module;
The identification module for analyzing the classroom image, and judges abnormal behavior in the classroom image It is raw;The identification module includes:
Behavioral value unit, for using trained depth tensor row network model come to the student in the classroom image into Row behavioral value;
The reminding module, for the recognition result of the identification module to be notified teacher.
11. a kind of teaching auxiliary system as claimed in claim 10, which is characterized in that be additionally provided with:With the identification module The memory module of connection;The memory module achieves the recognition result and goes forward side by side edlin analysis for synchronous;
The identification module further includes:
Expression Recognition unit, for using trained depth tensor row network model come to the student in the classroom image into Row Expression Recognition;
The Expression Recognition unit includes Face datection subelement and convolutional neural networks grader.
CN201710667590.7A 2017-08-07 2017-08-07 A kind of teaching auxiliary and the teaching auxiliary system using this method Pending CN108073888A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710667590.7A CN108073888A (en) 2017-08-07 2017-08-07 A kind of teaching auxiliary and the teaching auxiliary system using this method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710667590.7A CN108073888A (en) 2017-08-07 2017-08-07 A kind of teaching auxiliary and the teaching auxiliary system using this method

Publications (1)

Publication Number Publication Date
CN108073888A true CN108073888A (en) 2018-05-25

Family

ID=62159440

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710667590.7A Pending CN108073888A (en) 2017-08-07 2017-08-07 A kind of teaching auxiliary and the teaching auxiliary system using this method

Country Status (1)

Country Link
CN (1) CN108073888A (en)

Cited By (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830207A (en) * 2018-06-06 2018-11-16 成都邑教云信息技术有限公司 A kind of Internet education warning system
CN108921748A (en) * 2018-07-17 2018-11-30 郑州大学体育学院 Didactic code method and computer-readable medium based on big data analysis
CN108960101A (en) * 2018-06-22 2018-12-07 张小勇 Data processing method and system
CN109165633A (en) * 2018-09-21 2019-01-08 上海健坤教育科技有限公司 A kind of intelligent interactive learning system based on camera perception
CN109359606A (en) * 2018-10-24 2019-02-19 江苏君英天达人工智能研究院有限公司 A kind of classroom real-time monitoring and assessment system and its working method, creation method
CN109377432A (en) * 2018-12-21 2019-02-22 广东粤众互联信息技术有限公司 A kind of tutoring system based on big data acquisition
CN109522883A (en) * 2018-12-28 2019-03-26 广州海昇计算机科技有限公司 A kind of method for detecting human face, system, device and storage medium
CN109977989A (en) * 2019-01-17 2019-07-05 北京工业大学 A kind of processing method of image tensor data
CN110009539A (en) * 2019-04-12 2019-07-12 烟台工程职业技术学院(烟台市技师学院) A kind of student is in school learning state smart profile system and application method
CN110175534A (en) * 2019-05-08 2019-08-27 长春师范大学 Teaching assisting system based on multitask concatenated convolutional neural network
CN110175501A (en) * 2019-03-28 2019-08-27 重庆电政信息科技有限公司 More people's scene focus recognition methods based on recognition of face
CN110363245A (en) * 2019-07-17 2019-10-22 上海掌学教育科技有限公司 Excellent picture screening technique, the apparatus and system of Online class
CN110414415A (en) * 2019-07-24 2019-11-05 北京理工大学 Human bodys' response method towards classroom scene
CN110827491A (en) * 2019-09-26 2020-02-21 天津市华软创新科技有限公司 School student behavior big data analysis system
CN110827595A (en) * 2019-12-12 2020-02-21 广州三人行壹佰教育科技有限公司 Interaction method and device in virtual teaching and computer storage medium
CN111339809A (en) * 2018-12-20 2020-06-26 深圳市鸿合创新信息技术有限责任公司 Classroom behavior analysis method and device and electronic equipment
CN111832595A (en) * 2019-04-23 2020-10-27 北京新唐思创教育科技有限公司 Teacher style determination method and computer storage medium
WO2020216286A1 (en) * 2019-04-23 2020-10-29 北京新唐思创教育科技有限公司 Method for training teaching style prediction model, and computer storage medium
CN112116181A (en) * 2019-06-20 2020-12-22 北京新唐思创教育科技有限公司 Classroom quality model training method, classroom quality evaluation method and classroom quality evaluation device
CN112201116A (en) * 2020-09-29 2021-01-08 深圳市优必选科技股份有限公司 Logic board identification method and device and terminal equipment
WO2021047185A1 (en) * 2019-09-12 2021-03-18 深圳壹账通智能科技有限公司 Monitoring method and apparatus based on facial recognition, and storage medium and computer device
CN112597977A (en) * 2021-03-02 2021-04-02 南京泛在实境科技有限公司 HSV-YOLOv 3-based online class student behavior identification method
CN114897647A (en) * 2022-04-27 2022-08-12 合创智能家具(广东)有限公司 Teaching auxiliary system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110169932A1 (en) * 2010-01-06 2011-07-14 Clear View Technologies Inc. Wireless Facial Recognition
CN103258204A (en) * 2012-02-21 2013-08-21 中国科学院心理研究所 Automatic micro-expression recognition method based on Gabor features and edge orientation histogram (EOH) features
CN106097790A (en) * 2016-08-31 2016-11-09 王翠丽 A kind of teaching auxiliary device
CN106778745A (en) * 2016-12-23 2017-05-31 深圳先进技术研究院 A kind of licence plate recognition method and device, user equipment

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20110169932A1 (en) * 2010-01-06 2011-07-14 Clear View Technologies Inc. Wireless Facial Recognition
CN103258204A (en) * 2012-02-21 2013-08-21 中国科学院心理研究所 Automatic micro-expression recognition method based on Gabor features and edge orientation histogram (EOH) features
CN106097790A (en) * 2016-08-31 2016-11-09 王翠丽 A kind of teaching auxiliary device
CN106778745A (en) * 2016-12-23 2017-05-31 深圳先进技术研究院 A kind of licence plate recognition method and device, user equipment

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
胡勇等: "复杂掩模图数据处理与转换的研究", 《计算机技术与应用》 *

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108830207A (en) * 2018-06-06 2018-11-16 成都邑教云信息技术有限公司 A kind of Internet education warning system
CN108960101B (en) * 2018-06-22 2022-06-14 张小勇 Data processing method and system
CN108960101A (en) * 2018-06-22 2018-12-07 张小勇 Data processing method and system
CN108921748A (en) * 2018-07-17 2018-11-30 郑州大学体育学院 Didactic code method and computer-readable medium based on big data analysis
CN108921748B (en) * 2018-07-17 2022-02-01 郑州大学体育学院 Teaching planning method based on big data analysis and computer readable medium
CN109165633A (en) * 2018-09-21 2019-01-08 上海健坤教育科技有限公司 A kind of intelligent interactive learning system based on camera perception
CN109359606A (en) * 2018-10-24 2019-02-19 江苏君英天达人工智能研究院有限公司 A kind of classroom real-time monitoring and assessment system and its working method, creation method
WO2020082971A1 (en) * 2018-10-24 2020-04-30 江苏君英天达人工智能研究院有限公司 Real-time classroom monitoring and evaluation system and operation and creation method thereof
CN111339809A (en) * 2018-12-20 2020-06-26 深圳市鸿合创新信息技术有限责任公司 Classroom behavior analysis method and device and electronic equipment
CN109377432A (en) * 2018-12-21 2019-02-22 广东粤众互联信息技术有限公司 A kind of tutoring system based on big data acquisition
CN109522883A (en) * 2018-12-28 2019-03-26 广州海昇计算机科技有限公司 A kind of method for detecting human face, system, device and storage medium
CN109977989A (en) * 2019-01-17 2019-07-05 北京工业大学 A kind of processing method of image tensor data
CN109977989B (en) * 2019-01-17 2021-04-20 北京工业大学 Image tensor data processing method
CN110175501A (en) * 2019-03-28 2019-08-27 重庆电政信息科技有限公司 More people's scene focus recognition methods based on recognition of face
CN110009539A (en) * 2019-04-12 2019-07-12 烟台工程职业技术学院(烟台市技师学院) A kind of student is in school learning state smart profile system and application method
CN111832595A (en) * 2019-04-23 2020-10-27 北京新唐思创教育科技有限公司 Teacher style determination method and computer storage medium
WO2020216286A1 (en) * 2019-04-23 2020-10-29 北京新唐思创教育科技有限公司 Method for training teaching style prediction model, and computer storage medium
CN110175534A (en) * 2019-05-08 2019-08-27 长春师范大学 Teaching assisting system based on multitask concatenated convolutional neural network
CN112116181A (en) * 2019-06-20 2020-12-22 北京新唐思创教育科技有限公司 Classroom quality model training method, classroom quality evaluation method and classroom quality evaluation device
CN110363245A (en) * 2019-07-17 2019-10-22 上海掌学教育科技有限公司 Excellent picture screening technique, the apparatus and system of Online class
CN110363245B (en) * 2019-07-17 2023-05-12 上海掌学教育科技有限公司 Online classroom highlight screening method, device and system
CN110414415A (en) * 2019-07-24 2019-11-05 北京理工大学 Human bodys' response method towards classroom scene
WO2021047185A1 (en) * 2019-09-12 2021-03-18 深圳壹账通智能科技有限公司 Monitoring method and apparatus based on facial recognition, and storage medium and computer device
CN110827491A (en) * 2019-09-26 2020-02-21 天津市华软创新科技有限公司 School student behavior big data analysis system
CN110827595A (en) * 2019-12-12 2020-02-21 广州三人行壹佰教育科技有限公司 Interaction method and device in virtual teaching and computer storage medium
CN112201116A (en) * 2020-09-29 2021-01-08 深圳市优必选科技股份有限公司 Logic board identification method and device and terminal equipment
CN112597977A (en) * 2021-03-02 2021-04-02 南京泛在实境科技有限公司 HSV-YOLOv 3-based online class student behavior identification method
CN114897647A (en) * 2022-04-27 2022-08-12 合创智能家具(广东)有限公司 Teaching auxiliary system

Similar Documents

Publication Publication Date Title
CN108073888A (en) A kind of teaching auxiliary and the teaching auxiliary system using this method
US11270526B2 (en) Teaching assistance method and teaching assistance system using said method
CN113963445B (en) Pedestrian falling action recognition method and equipment based on gesture estimation
CN111898406B (en) Face detection method based on focus loss and multitask cascade
CN109271888A (en) Personal identification method, device, electronic equipment based on gait
CN108549876A (en) The sitting posture detecting method estimated based on target detection and human body attitude
CN109284737A (en) A kind of students ' behavior analysis and identifying system for wisdom classroom
CN108805070A (en) A kind of deep learning pedestrian detection method based on built-in terminal
CN106650619A (en) Human action recognition method
CN107016357A (en) A kind of video pedestrian detection method based on time-domain convolutional neural networks
US20210319363A1 (en) Method and system for generating annotated training data
CN106683091A (en) Target classification and attitude detection method based on depth convolution neural network
CN109255375A (en) Panoramic picture method for checking object based on deep learning
CN107092883A (en) Object identification method for tracing
CN111507227B (en) Multi-student individual segmentation and state autonomous identification method based on deep learning
CN111368768A (en) Human body key point-based employee gesture guidance detection method
CN107948586A (en) Trans-regional moving target detecting method and device based on video-splicing
Wu et al. A size-grading method of antler mushrooms using yolov5 and pspnet
Feng Mask RCNN-based single shot multibox detector for gesture recognition in physical education
CN106156713A (en) A kind of image processing method automatically monitored for examination hall behavior
Iren Comparison of yolov5 and yolov6 models for plant leaf disease detection
CN109118512A (en) A kind of classroom based on machine vision is come to work late and leave early detection method
Yin Albert et al. Identifying and Monitoring Students’ Classroom Learning Behavior Based on Multisource Information
CN115471773B (en) Intelligent classroom-oriented student tracking method and system
CN112686128B (en) Classroom desk detection method based on machine learning

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20180525

RJ01 Rejection of invention patent application after publication