CN111598453A - Control ergonomics analysis method, equipment and system based on executive force in virtual scene - Google Patents

Abstract

The invention discloses a control work efficiency analysis method, equipment and a system based on execution force in a virtual scene, wherein the method comprises the following steps: acquiring control result data generated by controlling a target object to execute a target task under a virtual scene by a control player, wherein the target object is the virtual object under the virtual scene; according to the control result data, obtaining task work efficiency scores of the control players for the target tasks; obtaining the control score of the control player according to the task work efficiency score; and executing a set operation according to the control score.

Description

Control ergonomics analysis method, equipment and system based on executive force in virtual scene

technical field

The invention relates to the technical field of automatic analysis of manipulation ergonomics, and more particularly, to a manipulation ergonomics analysis method, device and system based on executive force in a virtual scene.

Background technique

Different operators control the same target object to perform the target task, and will have different control ergonomics. For example, different operators control the same type of UAV to perform the same target task, and there will be different performances. Some operators can use a shorter time. Time to complete the target task, some operators have a good mental state when performing the target task, and so on. Analyzing the manipulation ergonomics of the operator when operating the target object to perform the target task can be used as the basis for selecting the operator who manipulates the target object, and can also be used as the basis for evaluating the degree of adaptation between any operator and any motion control device. At present, when analyzing the control ergonomics, experts are usually organized to manually score the operator to control the target object to perform the target task, so as to reflect the corresponding control ergonomics through the scoring results. The higher the score, the higher the control ergonomics. This manual scoring method not only consumes a lot of manpower, but also has the problems of low accuracy and unfairness due to excessive dependence on human subjective factors. Therefore, it is necessary to provide an intelligent solution for analyzing and controlling ergonomics .

SUMMARY OF THE INVENTION

An object of the embodiments of the present invention is to provide a new technical solution for analyzing manipulation ergonomics.

According to a first aspect of the present invention, there is provided a manipulation ergonomics analysis method based on executive force in a virtual scene, including:

Obtaining manipulation result data generated by the manipulation player manipulating a target object to perform a target task in a virtual scene, wherein the target object is a virtual object in the virtual scene;

According to the manipulation result data, obtain the task ergonomics score of the manipulation player for the target task;

Obtain the manipulation score of the manipulation player according to the task ergonomics score;

According to the manipulation score, a set operation is performed.

Optionally, the operation of performing the setting includes at least one of the following:

The first item is to output the manipulation score;

The second item is to provide the selection result of whether the manipulation player is selected according to the manipulation score;

The third item is to determine the manipulation level of the manipulation player according to the manipulation score;

Item 4: According to the manipulation score of the same manipulation player controlling the target object to perform the target task through different motion control devices, select a manipulation combination that makes the manipulation score meet the set requirements, wherein one manipulation combination includes a matching manipulation player. and motion controls.

Optionally, the method further includes:

In response to the operation of setting the application scene, provide a setting entry;

acquiring an application scenario input through the setting entry, wherein the input application scenario reflects an operation to be performed based on the manipulation score;

According to the input application scenario, the operation content of the set operation is determined.

Optionally, the method includes:

providing a configuration interface in response to an operation to configure the target task;

acquiring the configuration information for the target task input through the configuration interface;

According to the configuration information, the virtual scene corresponding to the target task is provided.

Optionally, the method further includes:

acquiring the control command generated by the manipulation player by manipulating the motion control device, and updating the virtual scene according to the control command;

Acquire feedback data generated by the virtual scene, and send the feedback data to the motion control device.

Optionally, obtaining the task ergonomics score of the manipulation player for the target task according to the manipulation result data includes:

According to the manipulation result data, obtain the individual scores of the operator for each set evaluation index;

According to each individual score and the weight coefficient of each evaluation index, the task ergonomics score of the manipulation player for the target task is obtained.

Optionally, the method further includes the step of obtaining the evaluation indicators, including:

Obtain the set initial evaluation index set;

Obtain the manipulation result data generated by the certified operator manipulating the target object to perform the target task, as the manipulation result reference data;

According to the manipulation result reference data and the set scoring rule, obtain the single item score of the certified operator for each index in the initial evaluation index set, as a single item reference score;

obtaining a correlation value representing the degree of correlation between each index in the initial evaluation index set and the manipulation level according to the known manipulation level of the certified operator and the single reference score;

According to the correlation value, each evaluation index is selected from the initial evaluation index set.

Optionally, the method further includes the step of obtaining the weight coefficient of each evaluation index, including:

Provide a weight comparison interface;

obtaining a comparison result of the importance of each two evaluation indicators input through the weight comparison interface;

Generate a judgment matrix according to the comparison result;

Based on the analytic hierarchy process algorithm, the weight coefficient of each evaluation index is obtained according to the judgment matrix.

Optionally, each evaluation index includes at least one index reflecting manipulation proficiency, at least one index reflecting position control capability, at least one index reflecting manipulation information processing capability, at least one index reflecting manipulation efficiency, and at least one index reflecting fault handling. at least one indicator of capability.

According to the second aspect of the present invention, there is also provided a manipulation ergonomics analysis device based on execution force in a virtual scene, the device comprising at least one computing device and at least one storage device, wherein the at least one storage device is used for analyzing the manipulation ergonomics. The apparatus is for storing instructions for controlling the at least one computing apparatus to perform the method according to the first aspect of the present invention.

According to a third aspect of the present invention, there is also provided a manipulation ergonomics analysis system based on execution force in a virtual scene, wherein the system includes a task execution device and the manipulation ergonomic analysis device according to the second aspect of the present invention, Wherein, the task execution device is connected in communication with the manipulation ergonomics analysis device.

Optionally, the motion control device of the task execution device is a flight control device, and the target object manipulated by the flight control device is an unmanned aerial vehicle in a virtual scene.

One beneficial effect of the embodiment of the present invention is that the method of the embodiment of the present invention provides a task ergonomics score indicating the completion of the target task by the manipulation player by manipulating the manipulation result data generated by manipulating the target object to perform the target task, thereby enabling at least The manipulation score of the manipulation player is determined according to the task ergonomics score, and selection of manipulation personnel for the target object, rating of manipulation personnel, and/or matching between manipulation personnel and motion control devices, etc. are performed. According to the method of this embodiment, the analysis of control ergonomics can be automatically completed, and labor cost and time cost can be saved. In addition, the analysis performed according to the method of this embodiment greatly reduces the dependence on expert experience and improves the accuracy of analysis and effectiveness.

Other features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the description serve to explain the principles of the invention.

FIG. 1 is a schematic diagram of the composition and structure of a manipulation ergonomics analysis system according to an embodiment;

2 is a schematic diagram of a hardware structure of a manipulation ergonomics analysis device according to an embodiment;

3 is a schematic flowchart of a manipulation ergonomics analysis method according to one embodiment;

4 is a schematic flowchart of a manipulation ergonomics analysis method according to another embodiment;

FIG. 5 is a schematic diagram of the composition and structure of an electromyography acquisition device according to an embodiment.

Detailed ways

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the invention unless specifically stated otherwise.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

<System Example>

FIG. 1 is a schematic structural diagram of an optional manipulation ergonomics analysis system 100 to which the method of the embodiment of the present invention can be applied.

As shown in FIG. 1 , the manipulation ergonomics analysis system 100 may include an electronic device 110 , a task execution device 120 and various physiological information collection devices 130 .

The electronic device 110 may be a server or a terminal device, which is not limited herein.

The server may be, for example, a blade server, a rack-mounted server, or the like, and the server may also be a server cluster deployed in the cloud, or the like. The terminal device may be any device with data processing capability, such as a PC, a notebook computer, and a tablet computer.

The electronic device 110 may include a processor 1101 , a memory 1102 , an interface device 1103 , a communication device 1104 , a display device 1105 , and an input device 1106 .

The memory 1102 is used to store computer instructions, and the memory 1102 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), non-volatile memory such as a hard disk, and the like. The processor 1101 is used to execute a computer program, and the computer program can be written using an instruction set of architectures such as x86, Arm, RISC, MIPS, SSE, and the like. The interface device 1103 includes, for example, various bus interfaces, for example, a serial bus interface (including a USB interface, etc.), a parallel bus interface, and the like. The communication device 1104 is capable of, for example, wired or wireless communication, for example, using at least one of an RJ45 module, a WIFI module, a 2G-6G mobile communication module, a network adapter of a Bluetooth module, and the like for communication. The display device 1105 is, for example, a liquid crystal display screen, an LED display screen, a touch display screen, or the like. The input device 1106 may include, for example, a touch screen, a keyboard, a mouse, and the like.

In this embodiment, the memory 1102 of the electronic device 110 is used to store computer instructions, and the computer instructions are used to control the processor 1101 to operate to implement the manipulation ergonomics analysis method according to any embodiment of the present invention. The skilled person can design the instruction according to the solution disclosed in the present invention. How the instruction controls the processor to operate is well known in the art, so it will not be described in detail here.

Although multiple devices of the electronic device 110 are shown in FIG. 1 , the present invention may only involve some of the devices. For example, the electronic device 110 only involves the memory 1102 , the processor 1101 , and the communication device 1104 .

In this embodiment, as shown in FIG. 1 , the task execution device 120 may be a task execution device based on hardware-in-the-loop simulation of a virtual environment, and the task execution device 120 may include a terminal device 1203 and a real motion control device 1201. The terminal The device 1203 is configured to provide a virtual scene corresponding to the target task, that is, a simulation scene. In this embodiment, the target object 1202 is also a virtual object in the virtual scene. In this embodiment, the motion control device 1201 is connected in communication with the terminal device 1203 to realize the interaction of data and/or commands between the motion control device 1201 and the virtual scene, so that the operator can control the virtual scene through the motion control device 1201 The target object 1202 performs the target task.

In this embodiment, the terminal device 1203 may have a hardware structure similar to that of the electronic device 110, which will not be repeated here. The terminal device 1203 and the electronic device 110 may be physically separated devices, or may be the same device, that is, The above virtual environment may also be provided by the electronic device 110, which is not limited herein.

In this embodiment, the operator can control the target object in the virtual scene through the motion control device 1201 to perform the target task. For example, the target object in the virtual scene may be a drone, and the motion control device 1201 is a flight control device for operating the drone. For another example, the target task includes completing at least one of figure-of-eight flight, spin flight, swarm flight, and the like under a set environment. For another example, the setting environment includes wind, rain, fog, and the like. Of course, the target object 1202 may also be other controlled objects, such as an unmanned vehicle, any type of robot, etc., which is not limited here.

The motion control device 1201 may include, for example, at least one of a remote control and a remote control handle.

The motion control device 1201 may include a processor, a memory, an interface device, an input device, a communication device, and the like. The memory can store computer instructions, and when the processor moves the instructions, it can perform: an operation of sending a corresponding control command to the terminal device 1203 according to the operation of the input device by the operator; obtaining the motion state data returned by the target object, and Perform corresponding processing operations; and upload the collected manipulation result data to the electronic device 110 , etc., which will not be further described here.

In this embodiment, the task execution device 120 is connected in communication with the electronic device 110 to upload the manipulation result data to the electronic device 110 . For example, this may be that the task performance device 120 is in communication with the electronic device 110 via the motion control device 1201 . For another example, this may also be that the motion control apparatus 1201 and the terminal device 1203 are both communicatively connected to the electronic device 110, which is not limited herein.

In FIG. 1 , each physiological information collection device 130 is used to provide the physiological information data required by the electronic device to implement the manipulation ergonomics analysis method according to any embodiment. Each physiological information collection device 130 is connected in communication with the electronic device 110 to upload the physiological information data provided by the electronic device 110 to the electronic device 110 .

Each physiological information collecting device 130 includes at least one of an EEG collecting device 1301 , an EMG collecting device 1302 , an ECG collecting device 1303 and a video collecting device 1304 for collecting facial expressions.

The physiological information data provided by the EEG acquisition device 1301 includes at least one of an EEG signal and an EEG image.

The physiological information data provided by the EMG acquisition device 1302 includes at least one of an EMG signal and an EMG image.

The physiological information data provided by the ECG acquisition device 1303 includes at least one of an ECG signal and an ECG image.

The physiological information data provided by the video capture device 1304 may include at least one of facial feature change data and facial image data.

Any physiological information collection device 130 may include a front-end collection device and a data processing circuit connected to the collection device. The front-end collection device is used to collect raw data, which may be an electrode device in contact with the control player, and the data processing circuit is used to analyze the raw data. The data is subjected to corresponding preprocessing, and the preprocessing includes at least one of signal amplification, filtering, denoising, and notch processing. The data processing circuit can be realized by a basic circuit built by electronic components, or by a processor. The implementation of the running instruction can also be implemented by a combination of the two, which is not limited here.

In the embodiment in which the cognitive neuroergonomics of the control player is not required, the system 100 may not include each physiological information collection device 130 .

The above electronic device 110 and the task execution device 120 and the electronic device 110 and each physiological information collection device 130 may be connected through wired or wireless communication, which is not limited herein.

In one embodiment, as shown in FIG. 2, the present invention provides a manipulation ergonomics analysis device 140 comprising at least one computing device 1401 and at least one storage device 1402, wherein the at least one storage device 1401 is used to store instructions, The instruction is used to control at least one computing device 1402 to execute the manipulation ergonomics analysis method based on the execution force in the virtual scene according to any embodiment of the present invention. The manipulation ergonomics analysis device 140 may include at least one electronic device 110, and may also include a terminal device 1203, etc., which is not limited herein.

<Method Example>

FIG. 3 is a schematic flowchart of a manipulation ergonomics analysis method according to an embodiment, and the method may be implemented by, for example, the manipulation ergonomic analysis device 140 shown in FIG. 2 . In this embodiment, the manipulation ergonomics analysis method of this embodiment is described by taking the analysis of a manipulation player performing a target task through a task execution device as an example, and the method may include the following steps S310-S340:

In step S310, the manipulation result data generated by the manipulation player manipulating the target object to perform the target task in the virtual scene is obtained.

In this embodiment, the manipulation result data may be provided by the task execution device 120, or the task execution device 120 may provide the manipulation ergonomics analysis device 140 with basic data for calculating the manipulation result data, and the manipulation ergonomics analysis device 140 may provide the manipulation result data according to the data. The manipulation result data is obtained by calculating the basic data for acquisition in this step S310.

The manipulation result data may include, for example, index data reflecting manipulation proficiency, index data reflecting position control capability, index data reflecting cluster manipulation information processing capability, index data reflecting cluster manipulation efficiency, and index data reflecting fault handling capability. of at least one type of data.

The above-mentioned index data reflecting the manipulation proficiency may include, for example, at least one item of command reflection delay time, manipulation delay time, and the number of task pauses. The index data reflecting the manipulation proficiency may be determined by the motion control device 1201 as shown in FIG. 1 .

The above-mentioned index data reflecting the position control capability may include, for example, at least one item of altitude deviation, horizontal deviation, heading deviation, and stability. The index data reflecting the position control capability can be derived from each sensor of the target object.

The above-mentioned index data reflecting the cluster manipulation information processing capability may include, for example, at least one item of cluster information reporting speed, cluster fault discovery speed, and the like. The index data reflecting the manipulation proficiency may be determined by the motion control device 1201 as shown in FIG. 1 .

The above-mentioned index data reflecting cluster manipulation efficiency may include, for example, at least one of a parameter value reflecting formation setting capability, a parameter value reflecting route planning capability, and the like. The parameter value of the reaction formation setting capability may include, for example, the time used for formation setting, the length of time for the corresponding formation to complete the target task, and the like. The parameter value of the reactive route planning capability may include, for example, the time required for route planning, the length of time to complete the target task, and the like.

The above-mentioned index data reflecting the fault handling capability may include, for example, at least one of a parameter value reflecting the emergency response capability, a parameter value reflecting the correctness of the fault handling, and the like. The parameter value of the emergency response capability may include, for example, the response time of the emergency response, the result of whether the emergency response is successful, and the like. The parameter value reflecting the correctness of the fault handling may include, for example, a result of whether the fault handling is successful or not.

The target object may be, for example, a drone or the like.

The target task includes the task content and the corresponding task environment.

In this embodiment, as shown in FIG. 1 , the manipulation player can manipulate the target object in the virtual scene provided by the terminal device 1203 through the motion control device 1201 , that is, the target object and the task environment are both virtual. In this embodiment, in order to realize the interaction of data and commands between the motion control device 1201 and the virtual scene, the method may further include the following steps S3011-S3013:

Step S3011, providing a virtual scene corresponding to the target task, wherein the target object is a virtual object in the virtual scene.

In step S3012, a control command generated by the operator by operating the motion control device 1201 is acquired, and the virtual scene is updated according to the control command.

In this step S3012, updating the virtual scene includes updating the task environment and the state of the target object, and the state includes the position and posture of the target object.

Step S3013 , acquiring feedback data generated by the virtual scene, and sending the feedback data to the motion control device 1201 .

The virtual scene includes all virtual things provided by the terminal device 1203 corresponding to the target task, including virtual environments and virtual objects.

In step S3013, the feedback data is collected by the virtual sensor of the target object, and sent by the device 140 to the motion control device 1201 for the manipulation player to make manipulation judgment. The feedback data can also be used by the device 140 to obtain at least part of the above-mentioned manipulation result data.

In this embodiment, the method may further include the following steps S3021-S3023:

Step S3021, providing a configuration interface in response to the operation of configuring the target task.

A simulation application may be installed on the device 140, and the interface of the simulation application may provide an entry for triggering the operation of configuring the target task, through which the configuration personnel can enter the configuration interface, and the configuration interface is provided by the configuration interface.

The configuration interface may include at least one form of an input box, a check box, and a drop-down list, so that the configuration personnel can configure the target task.

Step S3022: Obtain the configuration information for the target task input through the configuration interface.

In this step S3022, the configuration information input through the configuration interface may be acquired in response to the operation of completing the configuration. The configuration information includes, for example, information reflecting task content and task environment, and the like.

In this step S3022, the configuration personnel may, for example, trigger the operation of completing the configuration through buttons such as "confirm" or "submit" provided on the configuration interface.

Step S3023, according to the configuration information, provide a virtual scene corresponding to the target task.

The virtual scene includes virtual objects and virtual environments corresponding to the target task.

According to the above steps S3021-S3023, the configuration personnel can flexibly configure target tasks through the configuration interface as required, so that virtual scenes corresponding to different target tasks can be provided through the device 140.

In step S320, according to the manipulation result data obtained in step S310, the task ergonomics score of the manipulation player for the target task is obtained.

The task ergonomics score reflects the control player's ability to complete the target task.

In this embodiment, various evaluation indexes for evaluating task work efficiency may be set, so as to measure the ability of the manipulation player to complete the target task through each evaluation index.

Each evaluation index can be preset. Each evaluation index may also use a correlation analysis method to select at least some of the indexes that are highly correlated with the rating of the control player from the preset initial evaluation index set to form each final evaluation index.

Each evaluation index includes, for example, at least one of: an index reflecting manipulation proficiency, an index reflecting position control capability, an index reflecting cluster manipulation information processing capability, an index reflecting cluster manipulation efficiency, and an index reflecting fault handling capability.

For example, the index reflecting the position control capability may further include at least one of an altitude deviation index, a horizontal deviation index, a heading deviation index, and a stability index. For example, the index reflecting the cluster manipulation information processing capability may further include at least one of a cluster information reporting speed index, a cluster fault finding speed index, and the like. For example, the index reflecting the cluster manipulation efficiency may further include at least one of a formation setting capability index, a route planning capability index, and the like. For example, the index reflecting the fault handling capability may further include at least one of an index reflecting the emergency response capability, an index reflecting the correctness of the fault handling, and the like.

In this embodiment, a comparison table reflecting the mapping relationship between the manipulation result data and the individual scores of each evaluation index may be preset, so as to obtain the operator's response to each evaluation index according to the manipulation result data obtained in step S310 and the comparison table single item score.

Since the manipulation result data includes the index data corresponding to each evaluation index, the comparison table may include the mapping relationship between the data range of each index data and the single score of the corresponding evaluation index. In this way, according to the comparison table, it is possible to According to the manipulation result data of any manipulation player, the individual scores of the manipulation player for each evaluation index are obtained.

The single item score can be represented by a score from 1 to 10 or 1 to 100, or by a numerical value representing a grade of excellent, good, medium, and poor, which is not limited here.

In this regard, in one embodiment, obtaining the task ergonomics score of the manipulation player for the target task according to the manipulation result data in step S320 may include: obtaining, according to the manipulation result data, the operator's score for each set evaluation index. single item score; and, according to each single item score and the respective weight coefficients of each evaluation index, obtain the task ergonomics score of the control player for the target task.

In this embodiment, the sum of the respective weight coefficients of each evaluation index is equal to 1. The respective weight coefficients of each evaluation index may be preset, or may be obtained by the AHP, etc., which are not limited here.

In this embodiment, the task ergonomics score Q ₁ can be obtained by the following formula (1):

In formula (1), i represents the ith evaluation index, _wi is the weight coefficient of the _ith evaluation index, qi is the individual score of the control player for the ith evaluation index, and M is the total number of each evaluation index.

Step S330, obtaining the manipulation score of the manipulation player according to the task ergonomics score.

In this embodiment, the task ergonomics score can be directly used as the manipulation score of the manipulation player.

In this embodiment, the manipulation score of the manipulation player may also be obtained in combination with other scores of the manipulation player.

For example, the other score may include a cognitive neuroergonomics score determined for cognitive abilities exhibited during manipulation. For another example, the other scores may also include daily performance scores and the like.

In step S340, the set operation is performed according to the manipulation score obtained in step S330.

In one embodiment, performing the set operation in step S340 may include the first operation, that is, outputting the manipulation score.

Outputting the manipulation score may include: driving a display device of the device 140 or a display device connected to the device 140 to display the manipulation score.

Outputting the manipulation score may also include: sending the manipulation score to the terminal device registered by the user who customized the manipulation score, or to the user account of the user who customized the manipulation score.

The user is, for example, a manipulation rater, and the user can register the device information of the terminal device with the device 140, so that the device 140 can send the manipulation score to the terminal device after obtaining the manipulation score of the manipulation player.

In the case of developing a manipulation analysis application for the method of this embodiment, the manipulation rating personnel can install the client of the application on their own terminal device, and obtain the manipulation score of the manipulation player by logging in to the user account registered in the application Wait.

The terminal device is, for example, a PC, a notebook computer, or a mobile phone, which is not limited herein.

In one embodiment, the setting operation performed in step S340 may include the second operation, that is, according to the manipulation score, providing a selection result of whether the manipulation player is selected. According to this embodiment, selection of the operator can be realized. Here, a score threshold may be set, and when the manipulation score is higher than or equal to the score threshold, it is determined that the manipulation player is selected. In this embodiment, performing the setting operation may further include: outputting the selection result in an arbitrary manner. The arbitrary manner includes displaying, printing, sending, and the like.

In one embodiment, the set operation performed in step S340 may include a third operation, that is, determining the manipulation level of the manipulation player according to the manipulation score. Here, a comparison table reflecting the correspondence between the manipulation score and the manipulation level may be preset, so as to determine the manipulation level of the corresponding manipulation player according to the manipulation score of any manipulation player and the comparison table. In this embodiment, performing the set operation may further include: outputting the manipulation level in an arbitrary manner.

In one embodiment, the operation of performing the setting in step S340 may include the fourth operation, that is, according to the manipulation scores of the same manipulation player manipulating the target object to perform the target task through different motion control devices, select the manipulation score that satisfies the setting. A desired control combination, wherein a control combination includes a suitable control player and a motion control device. In this embodiment, performing the set operation may further include: outputting the manipulation combination in an arbitrary manner.

In this embodiment, since the same manipulation player has different proficiency levels for different motion control devices, in this example, not only a manipulation combination that makes the manipulation score meet the set requirements can be selected, but also a manipulation combination that is most suitable for the manipulation player can be obtained. motion control device. In this example, the setting requirement is, for example, that the manipulation score is greater than or equal to the setting value, or the like.

In one embodiment, the user may be allowed to select the operation to be performed in step S460, therefore, the method may further include: in response to the operation of setting the application scene, providing a setting entry; acquiring the application scene input through the setting entry, wherein , the application scenario reflects the operation to be performed based on the manipulation score; and, according to the input application scenario, the operation content of the set operation is determined.

For example, according to the input application scenario, it is determined that the operation content of the above-mentioned set operation includes at least one operation among the above-mentioned operations.

According to the above steps S310 to S340, the method of this embodiment can determine the manipulation score for the manipulation player according to the manipulation result data generated by the manipulation player manipulating the target object to perform the target task, which will save labor costs and time to a great extent. Cost, and greatly reduce the dependence on expert experience, improve the accuracy and validity of the analysis. In addition, the manipulation score can be used by the relevant personnel to select the operator, perform the rating of the operator, and/or perform matching settings between the operator and the motion control device, and the like.

According to the above steps S310-S340, the method of this embodiment completes the analysis of the manipulation player by means of semi-physical simulation in a virtual scene, without the need to accurately manipulate the venue and target object, which greatly reduces the cost of manipulation analysis.

FIG. 4 is a schematic flowchart of a manipulation ergonomics analysis method according to another embodiment. In this embodiment, the manipulation score of the manipulation player is determined based on the task ergonomics score and the cognitive neural ergonomics score. The method can be implemented, for example, by the manipulation ergonomics analysis device 140 as shown in FIG. 2 . The method may include the following steps S410-S460:

In step S410, the manipulation result data generated by the manipulation player manipulating the target object to perform the target task in the virtual scene is obtained.

In step S420, according to the manipulation result data obtained in step S420, the task ergonomics score of the manipulation player for the target task is obtained.

Step S430, acquiring physiological state data generated by the manipulation player manipulating the target object to perform the target task in the virtual scene.

The physiological state data reflects the cognitive ability of the control player for the target task. The stronger the cognitive ability, the easier the control player is to complete the target task; the weaker the cognitive ability, the harder the control player is to complete the target task, and the goal The difficulty of the task will have corresponding responses in the physiological state of the manipulating players, for example, the response of heart rate, the response of EEG, the response of EMG, the response of facial expressions, etc. Therefore, in this embodiment, according to the physiological state data, a cognitive neural ergonomics score reflecting the cognitive ability of the control player for the target task can be obtained. The higher the cognitive neural ergonomics score, the stronger the control player has Ability to perform the target task.

The physiological state data is multidimensional data including a plurality of index data. The physiological state data may include, for example, index data reflecting attention, index data reflecting brain load, index data reflecting nerve fatigue, index data reflecting muscle fatigue, and index data reflecting emotional control ability. one.

Correspondingly, each physiological characteristic index used to evaluate the cognitive neural ergonomics of the control player includes, for example, at least one of: attention index, brain load index, executive ability index, nerve fatigue index, muscle fatigue index and emotion control index. According to the physiological state data, index data corresponding to each physiological characteristic index can be obtained.

The physiological state data can be determined according to the physiological information data provided by each physiological information collection device. Therefore, in one embodiment, obtaining the physiological state data generated by the manipulation player manipulating the target object to perform the target task in step S430 may include: obtaining the physiological information data provided by each physiological information collecting device; and, according to the physiological information data to obtain the set physiological state data.

In this embodiment, the physiological information data provided by any physiological information acquisition device may include at least one of physiological signal data and physiological image data.

For example, each physiological information acquisition device includes an EEG acquisition device 1301 as shown in FIG. 1 , and the physiological information data provided by the EEG acquisition device 1301 may include at least one of an EEG signal (electrical signal) and an EEG image.

For another example, each physiological information collection device includes an EMG collection device 1302 as shown in FIG. 1 , and the physiological information data provided by the EMG collection device 1302 may include at least one of an EMG signal (electrical signal) and an EMG image .

For another example, each physiological information collection device includes an ECG collection device 1303 as shown in FIG. 1 , and the physiological information data provided by the ECG collection device 1303 may include at least one of an ECG signal and an ECG image.

For another example, each physiological information collection device includes a video collection device 1304 as shown in FIG. 1 , and the physiological information data provided by the video collection device 1304 includes at least one of facial feature change data and facial image data. The change data of the facial feature, for example, includes at least one of data of occurrence of blinking action and data of occurrence of yawning action.

After any physiological information collection device collects the original data through the front-end collection device, it can perform at least one of signal amplification, filtering, denoising, and notch processing on the original data, and generate the physiological information data to provide to the device 140. , so that the device 140 can obtain the set physiological state data according to the physiological information data.

According to the natural connection between the physiological characteristic index and the physiological information data, and according to the physiological information data provided by the EEG acquisition device 1301, the index data of the psychological index characteristics such as the attention index, the brain load index and the nerve fatigue index of the control player can be obtained. According to the physiological information data provided by the myoelectricity collection device 1302, the index data of the psychological characteristic index such as the muscle fatigue degree index of the control player can be obtained. According to the physiological information data provided by the ECG collection device 1303, the index data of the psychological characteristic index such as the control player's ability to control emotion can be obtained. According to the physiological information data provided by the video capture device 1304, the index data of the psychological characteristic indicators such as the fatigue index of the control player can be obtained. The change in the number of blinks and/or the number of yawns of the control player can be determined, and whether there is an abnormal change in the number of blinks and/or the number of yawns can reflect the degree of fatigue of the control player. Therefore, the indicator data can be based on the control player. The fatigue level determined by the change in the number of blinks and/or the number of yawns by the competitor.

In this embodiment, the above-mentioned index data can be obtained according to the physiological information data provided by each physiological information collection device, and the index data can be determined according to each parameter value calculated from the physiological information data, and the parameter value is used for evaluating the corresponding physiological information. The value of the parameter of the feature indicator. For example, the attention index and the like can be evaluated by the variance of the relative power of the EEG signals.

Therefore, in one embodiment, obtaining the set physiological state data according to the physiological information data may include: obtaining, according to the physiological information data, a parameter value of a parameter used to evaluate the reflected physiological characteristic index; according to the parameter value , determining the index data of the control player for the corresponding physiological characteristic index; and, generating the physiological state data including all the determined index data.

For example, the variance of the relative power based on the power spectral density of the EEG signal is used to evaluate the attention index and the like.

In one embodiment, EEG images, EMG images, and electrocardiogram images can be directly used as part of the physiological state data to determine the cognitive neural ergonomic score of the control player.

Since the physiological information data comes from different physiological information collection devices, in order to make the evaluation of the cognitive ability of the control player based on the physiological information data have the same time reference, in one embodiment, each physiological information collection device provides The physiological information data may include: controlling each physiological information collecting device to perform the collecting operation synchronously; acquiring the physiological information data generated by each physiological information collecting device through the corresponding collecting operation.

In this embodiment, for example, by setting a unified clock reference, each physiological information collection device can be triggered to start and end a corresponding collection operation synchronously.

In step S440, according to the physiological state data obtained in step S430, the cognitive neural ergonomic score of the control player for the target task is obtained.

The cognitive neural ergonomic score reflects the cognitive ability of the manipulation player for the target task.

In one embodiment, the cognitive neural ergonomic score may be obtained with reference to the manner of obtaining the task function score, which will not be repeated here.

In another embodiment, the cognitive neural ergonomic score of the control player can also be obtained according to the physiological state data and the pre-trained cognitive neural ergonomic model, that is, the physiological state data can be input into the preset cognitive neural ergonomics In the ergonomic model, the cognitive neuroergonomics score was obtained.

In this embodiment, the cognitive neural ergonomic model reflects the mapping relationship between any physiological state data and the cognitive neural ergonomic score.

The cognitive neural ergonomic model can be obtained by training based on a deep convolutional neural network model, for example.

The deep convolutional neural network model includes an input layer, an output layer, and a hidden layer between the input layer and the output layer. The hidden layer can be composed of multi-layer neural networks, such as convolutional layers and pooling layers. The convolutional layer is mainly responsible for extracting the main features of the upper neural network. The pooling layer, also known as the feature mapping layer, is mainly responsible for mapping multiple features of the upper layer to a new plane.

In one embodiment, the method may further include the step of obtaining the cognitive neural ergonomic model, including: obtaining training samples, wherein each training sample corresponds to an operator, and each training sample includes the physiological state of the corresponding operator data and the known manipulation score of the corresponding operator; train a deep convolutional neural network model according to the training sample, and determine the network parameters of the model; and obtain the cognitive neural ergonomic model according to the network parameters.

In step S450, the manipulation score of the manipulation player is obtained according to the task ergonomics score and the cognitive neural ergonomics score.

In this embodiment, the sum of the task ergonomics score and the cognitive neural ergonomics score may be directly used as the manipulation score of the manipulation player.

In this embodiment, different weight coefficients can also be set for the task ergonomics score and the cognitive neural ergonomics score as required, so as to obtain the manipulation score of the control player according to the task ergonomics score, the cognitive neural ergonomics score, and their respective weight coefficients Q is shown in the following formula (2):

Q=α ₁ ×Q ₁ +α ₂ ×Q ₂ formula (2);

In formula (2), α ₁ is the weight coefficient of the task ergonomics score, Q ₁ is the task ergonomics score, α ₂ is the cognitive neural ergonomic score, and α ₂ is the weight coefficient of the cognitive neural ergonomic score.

In this embodiment, the sum of the weighting coefficients α ₁ and α ₂ is equal to 1. The weighting coefficients α ₁ and α ₂ can be preset. The value of the weight coefficients α ₁ and α ₂ is directly related to the selection requirements of the control players, and depends on the screening preference of the Standards Association for the actual task completion ability and potential cognitive ability of the control players, such as the preference for screening the actual task completion ability. If you prefer to select a controller with strong potential cognitive ability, you can set α ₁ to be smaller _{than α 2 .}

In this embodiment, the task ergonomics score and the cognitive neural ergonomics score may also be input into the support vector machine model, that is, the classification model, to obtain the manipulation score of the manipulation player.

The step of obtaining the support vector machine model may include: obtaining a plurality of training samples, each training sample corresponds to a manipulator, and each training sample includes the task ergonomics score, the cognitive neural ergonomics score and the corresponding manipulation score of the corresponding manipulation player and, using the training samples to train the model parameters of the support vector machine model, and then obtain the support vector machine model.

In step S460, the set operation is performed according to the manipulation score obtained in step S450.

According to the above steps S410 to S460, the method of this embodiment can determine the manipulation score for the manipulation player according to the manipulation result data and the psychological state data generated by the manipulation player manipulating the target object to perform the target task, which will be able to save to a great extent Labor costs and time costs are greatly reduced, and the dependence on expert experience is greatly reduced, and the accuracy and effectiveness of analysis are improved.

In one embodiment, the method may, according to a correlation analysis method, select an index that has a high correlation with the rating of the operator from a set of artificially set initial evaluation indicators, and form each final evaluation index, that is, form Various evaluation indicators can be used for determining the task ergonomics score in the above step S320, which is beneficial to improve the accuracy and effectiveness of the task ergonomics evaluation. The method of this embodiment may be implemented by the manipulation ergonomics analysis device 140 as shown in FIG. 2 .

In this embodiment, the method may further include steps S511 to S515 of obtaining each evaluation index that can be used in the above step S320:

Step S511, obtain the set initial evaluation index set.

Each index in the initial evaluation index set can be set by an expert. The initial evaluation index set includes, for example, all the above-mentioned indexes for task ergonomics and the like.

In this embodiment, the method may further include a step of acquiring an index set by an expert to form an initial evaluation index set, and the step may include, for example, providing a setting interface in response to an operation of setting the initial index; The initial index forms the initial evaluation index set.

Step S512 , obtaining the manipulation result data generated by the authenticated operator manipulating the target object to perform the target task, as manipulation result reference data.

For example, if the target object is a drone, the certified operator is an operator who has the qualification to operate the drone and has a known control level, such as an operator with an excellent level. Here, the excellent grades can be mapped to corresponding numerical values to facilitate the correlation calculation.

Step S513, according to the manipulation result reference data and the set scoring rule, obtain a single item score for each index in the initial evaluation index set by the certified operator as a single item reference score.

The set scoring rule reflects the mapping relationship between the data range of the index data of each index in the set and the corresponding single-item score.

In this step S513, according to the manipulation result reference data, the index data corresponding to each index in the set can be extracted, and then according to the scoring rule, the certified operator's score for each index in the initial evaluation index set can be obtained. Single item rating.

According to the step S513, a data set including each individual score of the certified operator and the qualification level of the certified operator can be obtained.

In this embodiment, a plurality of certified operators can be organized to participate in the implementation of the method, so as to obtain a plurality of such data sets. Screen the final evaluation indicators used to improve the accuracy of screening.

Step S514: Obtain a correlation value representing the correlation between each index in the initial evaluation index set and the manipulation level according to the manipulation level of the certified operator and the corresponding individual reference score.

In this step S514, the correlation value between each index in the initial evaluation index set and the control level can be obtained according to the data set of one or more operators. For example, there are 40 indicators in the initial evaluation index set. , then 40 correlation values are obtained.

In this step S514, a screening threshold may be set according to the value range of the correlation value, so as to screen each evaluation index to be finally used, that is, an index whose correlation value is greater than or equal to the screening threshold is selected from the initial evaluation index set. , as the final evaluation indicators.

In this embodiment, the correlation value between any index in the initial evaluation index set and the manipulation level can be obtained based on any correlation algorithm, so as to express the degree of correlation between the index and the manipulation level by the correlation value, for example, The correlation value can be obtained using the Pearson correlation coefficient method, ie, the correlation value is expressed by the Pearson correlation coefficient.

Step S515, according to the correlation value, select each evaluation index from the initial evaluation index set.

In this embodiment, taking the correlation value represented by the Pearson correlation coefficient as an example, the value range of the Pearson correlation coefficient is between -1 and 1, and an index such that the absolute value of the correlation value is greater than 0.5 can be selected as the final index. The various evaluation indicators used, etc.

In one embodiment, the method can obtain the respective weights of each evaluation index that is finally used through AHP, etc., so as to improve the accuracy of the obtained task ergonomics score. The method of this embodiment may be implemented by the manipulation ergonomics analysis device 140 as shown in FIG. 2 .

In this embodiment, the method further includes the step of obtaining the respective weights of each evaluation index, including the following steps S521-S524:

Step S521, providing a weight comparison interface.

In this embodiment, an expert may be organized to give a comparison result of the importance of each two evaluation indexes for each evaluation index. For example, if 20 evaluation indicators are selected in this embodiment, a 20×20 judgment matrix can be obtained according to the comparison result given by the expert.

In this step S521, a weight comparison interface may be provided, and the weight comparison interface provides an input box for giving a corresponding comparison result for every two evaluation indicators. In addition, the weight comparison interface can also provide a scale table as shown in Table 1 below, and the expert can give the corresponding comparison result in the input box according to the scale table for acquisition in the following step S521.

Table 1 Scale table

Scaling meaning 1 Indicates that the two indicators are of equal importance 3 Indicates that one of the two metrics is slightly more important than the other 5 Indicates that when comparing two metrics, one metric is significantly more important than the other 7 Indicates that when two metrics are compared, one metric is strongly important than the other 9 Indicates that one metric is extremely important compared to the other 2,4,6,8 The median of the above two adjacent judgments reciprocal The scale of index i compared with j is a_ij, then the scale of index j compared with i is a_ji=1/a_ij

Step S522: Obtain a comparison result of the importance of each two evaluation indexes input through the weight comparison interface.

Step S523, generating a judgment matrix according to the comparison result.

In step S523, the judgment matrix generated according to the comparison result is an M×M matrix.

Step S524, based on the AHP algorithm, obtain the respective weights of each evaluation index according to the judgment matrix.

In step S524, based on the AHP algorithm, the judgment matrix will be checked for consistency, so that the consistency of the judgment matrix is within an acceptable range, and if the judgment matrix fails the consistency check, the weights will be reloaded A comparison interface for experts to adjust the comparison results, wherein in the reloaded weight comparison interface, the previously input comparison results will be provided in the input box.

In this embodiment, the importance order of each evaluation index and the corresponding weight value can be obtained through the analytic hierarchy process, and then the respective weight of each evaluation index can be determined according to the importance order or the weight value.

<Example of EEG acquisition equipment>

In order to obtain the index data of physiological characteristic indexes such as attention index, brain load index, emotion control index and so on according to the physiological signal data provided by the EEG acquisition device 1301, the EEG acquisition device 1301 in this embodiment adopts broadband EEG The acquisition equipment is used to collect EEG signals of 0.5-100 Hz. The EEG signal has the characteristics of high low-frequency EEG energy and low high-frequency EEG energy. If the device's ability to suppress noise is poor, the collected broadband EEG will be submerged in noise. Therefore, the key to accurately obtaining the broadband information components of scalp EEG is how to improve the signal-to-noise ratio of EEG signals. In order to improve the signal-to-noise ratio value, in addition to the amplification, filtering and other operations performed by the hardware circuit, it is also necessary to remove the noise information generated by various disturbances in the EEG acquisition through software design. A processing method of physiological signal data provided by the EEG acquisition device 1301 is presented.

In this embodiment, the processing method may include: the EEG acquisition device 1301 sequentially filters the corresponding noise signals from the collected original EEG signals through a plurality of noise identification models to obtain denoised EEG signals; The physiological signal data includes the denoised EEG signal.

In this embodiment, the above multiple noise recognition models may include a noise recognition model for removing eye movement (EOG) interference, a noise recognition model for removing electrocardiogram (ECG) interference, and a noise recognition model for removing electromyography (EMG) interference , at least one item in a noise recognition model for removing head movement interference.

The processing method of this embodiment may be implemented by a processor (eg, MCU) of the EEG acquisition device 1301 , that is, the EEG acquisition device 1301 includes a processor and a memory, and the memory is used to store program instructions, and the instructions are used to control the processor Perform the above processing method.

The physiological signal data provided by the EEG acquisition device may include the denoised EEG signal, and may also include an EEG image and the like.

In this embodiment, the original EEG signal is collected by the front-end collection device of the EEG collection device, and the front-end collection device may be an electrode cap, and the electrode cap includes a cap body and a plurality of brain electrodes fixedly arranged on the cap body. In this way, by making the manipulation player wear the electrode cap, the original EEG signals of the manipulation player during the manipulation of the target object to perform the target task can be collected.

In this embodiment, since the electrode cap includes multiple groups of brain electrodes, multiple channels of original EEG signals can be collected through the electrode cap, thereby providing multiple channels of EEG signals, wherein one channel of EEG signal corresponds to one group of brain electrodes, Different brain electrodes correspond to different brain locations.

A group of brain electrodes may include a recording electrode (or referred to as an active electrode), a reference electrode and a ground electrode, and different groups of brain electrodes may share a reference electrode and/or a ground electrode and the like.

In this embodiment, since different positions of the brain have different sensitive matters, the position of the brain that is more sensitive to the set physiological characteristic index can be selected to set the recording electrode, and the EEG signal of the set channel can be used to set the recording electrode. Obtain index data for the corresponding physiological characteristic index.

For example, a set of brain electrodes including a recording electrode located in the left dorsolateral prefrontal region (corresponding to the F3 position in the 10-20 electrode placement system) can be used to determine the corresponding attention. Indicator data for the force indicator.

The original EEG signal may first undergo signal amplification, filtering, analog-to-digital conversion, etc., and then perform the operations of filtering corresponding noise signals through multiple noise identification models in the above sequence.

In one embodiment, the step of obtaining any noise identification model among the above multiple noise identification models may include: collecting a noise signal to be filtered out by the noise identification model; extracting signal characteristics of the noise signal; and, according to the signal feature, and the arbitrary noise recognition model is obtained by training.

In this embodiment, according to the noise signal to be filtered out, the target EEG signal with the noise signal and the reference EEG signal without the noise signal can be collected in a targeted manner, and the target EEG signal and the reference EEG signal are compared. By comparison, the noise signal to be filtered out can be obtained. When the noise identification model receives the input EEG signal, it can identify the components in the input EEG signal that conform to the characteristics of the signal, thereby achieving the purpose of removing the noise signal from the input EEG signal.

Taking the acquisition of a target EEG signal with noise signals generated by eye movement interference as an example, the target EEG signal can be collected by guiding the tester to blink and/or perform eye movements. For example, this can play a guiding animation on the computer screen used for the test, in which the squares are used to instruct the tester to blink, and the squares are used to guide the tester to make eye movements, etc., so that the target EEG signal can be collected.

Taking the acquisition of the target EEG signal with the noise signal of head movement interference as an example, the general head movement artifact is divided into two types: nodding and shaking head. Corresponding to the test of head movement interference, the target EEG signal with the noise signal of head movement interference is obtained. In addition, an acceleration sensor can also be used for head movement detection to extract head movement features and participate in the training of a noise recognition model for removing head movement interference.

Taking the acquisition of the target EEG signal with the noise signal with muscle interference as an example, the human facial muscles have varying degrees of influence on the EEG, and the brain electrodes located in the front are sensitive to facial muscles (corrugator and frontalis) Brain electrodes in the temporal lobe and middle are sensitive to the masticatory muscles (mainly masseter and temporalis), and acoustic stimulation can be used to guide the acquisition of this target EEG signal, and a continuous setting is used during tension and relaxation N100 evoked sound stimulation of a time length (eg, 100 ms) to acquire target EEG signals with noisy signals of muscle interference.

In this embodiment, according to the above test methods for different interferences, the tester can be guided to complete each test coherently, so as to improve the test efficiency. For example, this may be to first collect the reference EEG signal without any interference, and then collect the target EEG signal with the noise signal generated by eye movement interference, the target EEG signal with the noise signal caused by head movement interference, And the target EEG signal with the noise signal of muscle interference, etc.

In the training of the noise identification model, in order to make the noise identification model obtained by training have higher accuracy in the case of having fewer training samples, the characteristics of the extracted noise signal can be learned by means of transfer learning. For example, the inception family of models can be used for training. This can use a nonlinear system approach to convert the EEG signal into image segments one by one, and then use the inception model to learn them, etc.

Compared with the denoising method that removes the noise signal in the EEG signal through mathematical calculation, in this embodiment, the method of removing the set noise signal that may exist in the EEG signal by connecting multiple noise recognition models in series can better improve Accuracy and effectiveness of noise removal.

<Embodiment of EMG Collection Equipment>

FIG. 5 shows a schematic structural diagram of the electromyography acquisition device 1302 according to an embodiment.

As shown in FIG. 5 , the EMG collection device 1302 may include EMG collection electrodes 13021 , a filtering and signal amplification module 13022 , an analog-to-digital conversion module 13023 , a control module 13024 and a communication module 13025 , which are connected in sequence. The electromyography acquisition device 1302 may further include a power supply module 13026 , and the power supply module 13026 provides a working voltage for the analog-to-digital conversion module 13023 and the control module 13024 .

The EMG collecting electrode 13021 is in contact with the test player to collect the original EMG signal.

The filtering and signal amplifying module 13022 is used to perform analog filtering, signal amplification, notch preprocessing, etc. on the bioelectric signals accumulated on the collecting side of the EMG collecting electrode 13021 . The filtering and signal amplification module 13022 may include an analog band-pass filter circuit, an operational amplifier amplifier circuit and a notch preprocessing circuit.

The analog-to-digital conversion module 13023 can use an integrated AD conversion chip.

The control module 13024 may use an MCU chip or the like.

The communication module 13025 is configured to send the EMG signal output by the control module 13024 (ie, the EMG signal provided by the EMG acquisition device 1302 ) to the device 140 . The communication module 13025 may, for example, use a shielded USB communication cable for data transmission.

<Example of ECG Collection Device>

The ECG acquisition device 1303 may include sequentially connected acquisition electrodes, an analog filter module, a signal amplification module, a notch processing module, an analog-to-digital conversion module, a control module, and a communication module.

The human body's ECG beat frequency is 0.25~100Hz, and the concentration area of ECG signal energy is distributed in 0.25~35Hz. Therefore, the analog filter module can use a band-pass filter circuit, first perform high-pass filtering, and then perform second-order low-pass filtering. , and finally limit the signal frequency to 0.1Hz to 106Hz, and filter out other noise signals outside the bandwidth.

Since the ECG signal is a weak bioelectrical signal with a signal amplitude ranging from 0.05 to 4mv, it needs to be amplified before post-processing analysis can be performed. The signal amplification module can be used to amplify the output signal of the analog filter module. For example, the signal amplifying module can use an instrument operational amplifier chip to perform single-stage amplifying processing.

In order to better eliminate the interference of power frequency signals, 50Hz notch processing can be performed through the notch processing module.

The communication module is configured to send the ECG signal output by the control module (ie, the ECG signal provided by the ECG acquisition device 1303 ) to the device 140 .

The signal processing flow of the ECG acquisition device may include: the differential pair signal output by the acquisition electrode is processed by the analog filtering module, the signal amplification module, and the notch processing module in sequence, and the processed analog bioelectrical signal is input to the analog-to-digital conversion module. The processed analog bioelectrical signal is converted into a digital signal, and the analog-to-digital conversion module sends the digital signal to the control module for digital signal processing using transmission protocols such as SPI, and the communication module processes the digital signal (ie ECG acquisition). The EMG signal provided by the device) is sent to the device 140 .

<Example of Video Capture Device>

For video capture, the video capture device 1304 may include a camera and an image processing device connected to the camera.

Since the video file collected by the camera is composed of multiple frames of images, the video file can be divided into video images one by one. Therefore, the image processing device can divide the collected video file into video images for image processing.

This image processing includes face detection, ie, the identification of face regions from images captured by the camera.

This face detection can be done by scanning the image with a classifier. The classifier may be any existing face recognition classifier, for example, it may be a classifier trained by using the Adaboost algorithm, which will not be repeated here.

The image processing may further include: positioning and feature extraction of key points for the identified face region.

The positioning of the key points includes the positioning of the eyes, which can determine the difference threshold according to the difference between the eyes and the colors around the eyes, so as to determine the position of the human eye according to the change of the gray value for the binarized image. When the action of blinking occurs, the gray level of the area where the eye is located changes, and feature data representing the number of blinks can be extracted according to the change condition.

The location of the key point may also include the location of the mouth. After the position of the eyes is determined, the location of the mouth may be located according to the face model. When the test player yawns, the mouth opens, and the gray value of the mouth position changes. Therefore, the feature data representing the number of yawns can also be extracted according to the change of the gray value of the mouth position.

The physiological information data provided by the video capture device 1304 may include at least one of the above feature data representing the blinking action and the above feature data representing the yawning action.

In addition, the video capture device 1304 may also only perform image preprocessing for image binarization, and send the preprocessed image to the device 140, and the device 140 performs the above image processing operations, which are not limited herein.

In one embodiment, providing the physiological information data by the video capture device 1304 may include the following steps: acquiring a captured video image; identifying a face region in the video image; locating the eye position and Mouth position; according to the grayscale change of the eye position between each adjacent two frames of video images, determine the time point of the blinking action; according to the grayscale change of the mouth position between each adjacent two frames of video images, determine the occurrence of yawning The time point of the action; according to the above time point, the physiological information data provided by the video capture device is generated.

<Example of extracting index data>

In this embodiment, after receiving the physiological information data provided by each physiological information collection device 130, the device 140 will obtain the index data for the set physiological characteristic index according to the physiological information data.

For the EEG signal provided by the EEG acquisition device 1301, index data for physiological characteristic indexes such as attention index, brain load index, and nerve fatigue index can be extracted.

1) Regarding the extraction of index data for the attention index according to the EEG signal:

Attention refers to the ability of people's mental activities to point and focus on something. It includes a series of complex neural processing processes such as input, processing, integration, and adjustment and control of perceptual information. The basic premise.

During the manipulation process of the manipulation player manipulating the target object to perform the target task, the EEG acquisition device 1301 may provide the EEG signals generated by the manipulation player in the process.

In this embodiment, the index data for the attention index can be extracted according to the EEG signal corresponding to the first group of brain electrodes provided by the EEG acquisition device 130, which is recorded as the first EEG signal. Recording electrodes in the dorsolateral prefrontal region (corresponding to position F3 in the 10-20 electrode placement system).

In this embodiment, the first EEG signal is composed of multiple time series arranged in sequence, and the time length of each time series may be the same, for example, 10s. In this embodiment, the variance value of the relative power Rp of the first EEG signal in the β_1 frequency band (13Hz-20Hz) of a plurality of time series can be used to represent the concentration degree of the control player. The difference is used as the index data for the attention index, where the larger the Rp value, the higher the concentration of the operator in the corresponding time series.

The relative power Rp of the β_1 frequency band of any time series is: the ratio of the power density value of the β_1 frequency band of the corresponding time series to the power density value of the whole frequency band.

The calculation method of the power density value of any time series can be as follows: the time series X(n) can be divided into K sub-time series X1(n)~Xk(n). overlap each other and add equal-length Hamming windows to each sub-time series to avoid spectral leakage in the final result; compute the power density spectrum of each sub-time series; and, compute the average of the power density spectra of all sub-time series , as the power density value of the time series.

In this embodiment, according to the Rp value of each time series of the control player in the entire control process, the variance value of the RP value can be obtained, which is used to represent the fluctuation of the operator's attention during the completion of the target task. It means that the attention control ability is stronger. Finally, the index data of the player's attention index can be determined according to the variance value and the set classification threshold.

Taking the attention control ability into three grades: excellent, good and unqualified as an example, the classification threshold reflects the mapping relationship between the numerical range of the variance value and the grade, and the finally determined index data of the player's attention index is is the class you belong to, for example, the class you belong to is excellent. Here, different data identifiers can be used to represent different levels, for example, 100 means excellent, 101 means good, 110 means unqualified, etc., which is not limited here.

2) Regarding the extraction of the index data for the brain load index according to the EEG signal:

A classification model for classifying brain load is established in advance. For example, brain load is divided into three levels: high, medium, and low. Each level can be represented by a corresponding data identifier, that is, the extracted index data can be Corresponding brain load level.

Since the EEG in the prefrontal lobe is more sensitive to changes in brain load, the EEG signal provided by the EEG acquisition device 130 corresponding to the second group of brain electrodes located in the prefrontal lobe can be denoted as the second EEG signal and extracted. Indicator data for the brain load indicator.

In this embodiment, extracting the index data for the brain load index according to the EEG signal may include: obtaining a vector value of the second EEG signal for a set feature vector reflecting the brain load state; and inputting the vector value into the In the classification model, the index data for the brain load index is obtained.

The feature vector is composed of a plurality of features reflecting the state of brain load, and the second EEG signal constitutes the vector value for the value of each feature in the plurality of features, wherein each feature corresponds to one of the indicators used for evaluating the brain load parameter.

The features in the feature vector are divided into two categories, namely the first feature set and the second feature set. The first feature set includes: according to each of the multiple intrinsic mode functions (Intrinsic Mode Functions, IMFS) of the EEG signal The recursive feature quantities obtained by a function respectively.

The plurality of eigenmode functions are the first set number of eigenmode functions after the EEG signal is decomposed by Ensemble Empirical Mode Decomposition (EEMD), for example, the first 9 eigenmode functions.

The recursive feature quantities to be obtained according to each function may include recurrence ratio, determinism, laminarity, and entropy.

When the plurality of eigenmode functions are 9 eigenmode functions, and each eigenmode function corresponds to the above recursive feature quantities, the first feature set includes 36 features.

The second feature set includes: the relative powers of the EEG signals in five bands of 1-4 Hz, 4-8 Hz, 8-13 Hz, 13-30 Hz, and 30-45 Hz, ie, the second feature set may include 5 features.

In this embodiment, a training sample can be constructed by referring to the above method of extracting the vector value of the second EEG signal of the control player for the set feature vector, so as to obtain the classification model through training. For each training sample, a corresponding label can be set for each training sample according to the time taken by the tester to complete the test task,

3) Regarding the extraction of index data for nerve fatigue indicators according to EEG signals:

Since the P3a component in the EEG signal can reflect the state of brain fatigue, and the prefrontal lobe is easy to induce the P3a component, therefore, according to the EEG signal provided by the EEG acquisition device 130 corresponding to the second group of brain electrodes located in the prefrontal lobe, It is recorded as the second EEG signal, and the index data for the nerve fatigue index is extracted. The values of this indicator data include excellent, good and unqualified.

In this embodiment, a reference value of the amplitude of the P3a component can be set, and when the amplitude of the P3a component in the second EEG signal is greater than or equal to the set ratio of the reference value, the set ratio is higher than 50% , for example, 75%, it means that the control player did not experience nerve fatigue during the control process, the index data for the nerve fatigue index is excellent, and the amplitude of the P3a component in the second EEG signal is set at the beginning of the control. After a period of time, if it falls below the set ratio of the reference value, then its index data for the nerve fatigue index is good, and otherwise it is unqualified.

The set time length may be determined according to the time limit of the target task. Taking the time limit of 60 minutes as an example, the set time length may be 30 minutes or the like.

In one embodiment, during the period when the manipulation player manipulates the target object to perform the target task, the manipulation player can be stimulated in the auditory Oddball paradigm through the device 140, so that the P3a component appears in the EEG signal.

For the EMG signal provided by the EMG acquisition device 1302, the index data for the muscle fatigue index can be extracted.

Since the frequency information of the EMG signal (including the median frequency and the mean frequency) is negatively correlated with the fatigue level of the test players, the EMG signal has different frequency components under different fatigue levels. Therefore, the degree of muscle fatigue can be represented by the parameter values of the mean power frequency (MPF) and the median frequency (Median Frequency, MF).

In this embodiment, the collected electromyography signal can be analyzed through the FFT power spectrum scheme to obtain the average power frequency value and the median frequency value of the electromyography signal.

In this embodiment, the average power frequency value and the median frequency value can still be used to divide the index data for the muscle fatigue index into several levels, which will not be repeated here.

For the ECG signal provided by the ECG acquisition device 1303, the index data for the emotional control ability index can be extracted.

Since the heart rate is changing all the time, the change curve of the heart rate is naturally a line with ups and downs. The magnitude of heart rate variability directly reflects the regulatory effect of neurohumoral factors on cardiac beating. Analysis of heart rate variability from the regulation of neural activity essentially reflects the relationship between vagal activity and sympathetic activity and the balance between them. From the perspective of the time domain, the heart rate of the test players will change rapidly with different task events, which will affect the standard deviation (SDNN) value of the RR interval (abbreviated as NN interval) of all sinus beats. Therefore, SDNN can be used to quantitatively analyze the fluctuation value of emotional changes, and SDNN is positively correlated with emotional fluctuations, that is, the larger the SDNN value, the more significant the fluctuation of emotional fluctuations.

Therefore, the SDNN value can be determined according to the ECG signal to obtain the index data of the emotional control ability index of the manipulation player.

In this embodiment, the SDNN value can still be used to divide the index data of the emotion control ability index into several levels, which will not be repeated here.

For the psychological information data provided by the video capture device 1304, index data for the nerve fatigue index can be extracted.

As described above, the psychological information data provided by the video capture device 1304 may include the time point of blinking and the time point of yawning behavior.

According to the changes in the number of blinks and the yawning situation, it can be judged whether the test players are in a state of fatigue. For example, under normal conditions, the number of blinks of each person in one minute is not too different, but when a person is in a state of fatigue, the number of blinks will decrease or suddenly increase. Therefore, according to the number of blinks At the time point, compare the changes in the number of blinks within each set time length (for example, 1 minute). If the number of blinks in multiple consecutive time lengths is abnormal, it means that the user is in a fatigued state. For another example, in the normal control process, there is basically no yawning behavior, but when the control player is in a tired state, the yawning behavior will occur. If the yawning behavior occurs for several consecutive lengths of time, it can indicate that the control player is in a state of fatigue.

In this embodiment, the indicator data of whether the manipulation player is in a fatigue state can be given according to the psychological information data provided by the video collection device 1304 .

Each of the above embodiments focuses on describing the differences from other embodiments, and the same or similar aspects of different embodiments can be used with reference to each other.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims (10)

1. A control ergonomics analysis method based on execution force in a virtual scene comprises the following steps:

acquiring control result data generated by controlling a target object to execute a target task under a virtual scene by a control player, wherein the target object is the virtual object under the virtual scene;

according to the control result data, obtaining task work efficiency scores of the control players for the target tasks;

obtaining the control score of the control player according to the task work efficiency score;

executing a set operation according to the control score;

the obtaining of the task ergonomics score of the control player for the target task according to the control result data comprises:

according to the control result data, obtaining the single scores of the operator for the set evaluation indexes;

and obtaining task work efficiency scores of the control players for the target task according to the individual scores and the weight coefficients of the evaluation indexes.

2. The method of claim 1, wherein the performing the setting comprises at least one of:

a first item outputting the manipulation score;

a second item, which provides a selection result whether the control player is selected or not according to the control score;

a third item, determining the control level of the control player according to the control score;

and fourthly, selecting a control combination which enables the control score to meet the set requirement according to the control score of the same control player for controlling the target object to execute the target task through different motion control devices, wherein one control combination comprises the control player and the motion control device which are matched.

3. The method of claim 1, wherein the method further comprises:

providing a setting entrance in response to an operation of setting an application scene;

acquiring an application scene input through the setting entrance, wherein the input application scene reflects an operation to be executed based on a control score;

and determining the operation content of the set operation according to the input application scene.

4. The method of claim 1, wherein the method comprises:

providing a configuration interface in response to an operation to configure the target task;

acquiring configuration information for the target task input through the configuration interface;

and providing the virtual scene corresponding to the target task according to the configuration information.

5. The method of claim 1, wherein the method further comprises:

acquiring a control command generated by the control player through a control motion control device, and updating the virtual scene according to the control command;

and acquiring feedback data generated by the virtual scene, and sending the feedback data to the motion control device.

6. The method according to claim 1, wherein the method further comprises a step of obtaining the respective evaluation index, comprising:

acquiring a set initial evaluation index set;

acquiring control result data generated when an authenticated control person controls the target object to execute the target task, wherein the control result data is used as control result reference data;

obtaining a single score of each index in the initial evaluation index set by the authenticated operator according to the control result reference data and a set scoring rule, and using the single score as a single reference score;

obtaining a correlation value representing the correlation degree between each index in the initial evaluation index set and the control grade according to the known control grade of the authenticated control personnel and the single reference grade;

screening the evaluation indexes from the initial evaluation index set according to the correlation values; and/or the presence of a gas in the gas,

the method further comprises a step of obtaining a weight coefficient of each evaluation index, and the method comprises the following steps:

providing a weight comparison interface;

obtaining comparison results of the importance of each two evaluation indexes input through the weight comparison interface;

generating a judgment matrix according to the comparison result;

and based on a hierarchical analysis algorithm, obtaining the weight coefficient of each evaluation index according to the judgment matrix.

7. The method according to claim 1, wherein the evaluation indexes include at least one index reflecting a manipulation proficiency level, at least one index reflecting a position control ability, at least one index reflecting a manipulation information processing ability, at least one index reflecting a manipulation performance, and at least one index reflecting a fault processing ability.

8. A manipulation ergonomics apparatus based on execution forces in a virtual scene, comprising at least one computing device and at least one storage device, wherein,

the at least one storage device is to store instructions to control the at least one computing device to perform the method of any of claims 1 to 7.

9. A manipulation ergonomics system based on execution power in a virtual scene, wherein the system comprises a task execution device and the manipulation ergonomics device of claim 8, wherein the task execution device is communicatively connected to the manipulation ergonomics device.

10. The system of claim 9, wherein the motion control device of the task execution device is a flight control device, and the target object manipulated by the flight control device is a drone in a virtual scene.

Images