[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN116775438A - Model test method, device, electronic equipment and storage medium - Google Patents

Model test method, device, electronic equipment and storage medium Download PDF

Info

Publication number
CN116775438A
CN116775438A CN202310840748.1A CN202310840748A CN116775438A CN 116775438 A CN116775438 A CN 116775438A CN 202310840748 A CN202310840748 A CN 202310840748A CN 116775438 A CN116775438 A CN 116775438A
Authority
CN
China
Prior art keywords
model
test
data
performance
program
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310840748.1A
Other languages
Chinese (zh)
Inventor
朱悦
龚思思
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Netease Youdao Information Technology Beijing Co Ltd
Original Assignee
Netease Youdao Information Technology Beijing Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Netease Youdao Information Technology Beijing Co Ltd filed Critical Netease Youdao Information Technology Beijing Co Ltd
Priority to CN202310840748.1A priority Critical patent/CN116775438A/en
Publication of CN116775438A publication Critical patent/CN116775438A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3447Performance evaluation by modeling
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/3065Monitoring arrangements determined by the means or processing involved in reporting the monitored data
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/36Preventing errors by testing or debugging software
    • G06F11/3668Software testing
    • G06F11/3672Test management

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The embodiment of the application provides a model testing method, a device, electronic equipment and a storage medium. The model test method comprises the following steps: obtaining a model file and a test file, wherein the model file comprises an offline network model and a model execution file; the test file comprises model test data and a performance test program; the performance test program is operated, so that the performance test program can acquire performance evaluation data based on an offline network model, a model execution file and model test data; and forming a performance test report based on the performance evaluation data. According to the technical scheme, the testing efficiency of the offline network model can be improved, detailed testing index data can be obtained in real time, and the accurate optimization requirement of the offline network model is met.

Description

Model test method, device, electronic equipment and storage medium
Technical Field
Embodiments of the present application relate generally to the field of computers, and more particularly, to a model test method, apparatus, electronic device, and storage medium.
Background
This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein may include concepts that could be pursued, but are not necessarily ones that have been previously conceived or pursued. Accordingly, unless indicated otherwise, what is described in this section is not prior art to the description and claims of the present application and is not admitted to be prior art by inclusion in this section.
The offline network model, such as the offline TTS model and the offline ASR model, needs to be subjected to performance test after being iteratively updated to be applied to the terminal device. The existing test mode generally adopts a mode of testing based on an offline SDK. The test mode needs to be used for installing the Android Studio, configuring the Android SDK and the like, but the time spent for installing the Android Studio and configuring the Android SDK is long, the operation steps are complex, the using threshold is high, and blocking problems are frequently encountered in the debugging process, such as uninitialized TTS model display and the like. Furthermore, after each iteration update of the offline network model, operations such as Android Studio installation and Android SDK configuration need to be re-executed, so that a great deal of time is required to debug tools, and under the conditions of high-frequency update iteration of the offline network model and high-frequency release of terminal equipment, the test cost is high. Moreover, by adopting the test method, performance data cannot be obtained in real time to perform model tuning, so that problems are found out later, and meanwhile, test index data corresponding to each refinement business requirement cannot be obtained, and the optimization requirement of the model cannot be met.
In view of the foregoing, it is needed to provide an innovative model test method so as to improve the test efficiency of the offline network model, obtain detailed test index data in real time, and meet the requirement of accurate optimization of the offline network model.
Disclosure of Invention
In order to overcome the problems in the related art, it is desirable for embodiments of the present application to provide a model test method, apparatus, electronic device, and storage medium. The model test method can improve the test efficiency of the offline network model, acquire detailed test index data in real time, and meet the accurate optimization requirement of the offline network model.
In a first aspect of the embodiments of the present application, there is provided a model test method, including: obtaining a model file and a test file, wherein the model file comprises an offline network model and a model execution file; the test file comprises model test data and a performance test program; the performance test program is operated, so that the performance test program can acquire performance evaluation data based on an offline network model, a model execution file and model test data; and forming a performance test report based on the performance evaluation data.
In one embodiment of the application, the performance test program includes a test run program and a data acquisition program; the operation performance test program includes: running a test running program; and responding to the model operation command operation data acquisition program in the test operation program to obtain the performance evaluation data.
In one embodiment of the application, running the test run program includes: specifying operation parameters, wherein the operation parameters comprise parameters corresponding to an offline network model, parameters corresponding to model test data, parameters corresponding to terminal equipment and language parameters; executing a model operation command to enable the offline network model to start operation; recording the running starting time of the offline network model; when the operation time length of the offline network model reaches the preset time length, closing the operation process of the offline network model, and recording the operation ending time of the offline network model; and forming a model operation log based on the performance evaluation data, the operation starting time, the operation ending time and the operation parameters.
In one embodiment of the present application, after executing the model running command, further comprising: and enabling the model execution file to call the offline network model to operate in response to the model operation command.
In one embodiment of the present application, running the data acquisition program in response to the model running command in the test running program includes: responding to the model operation command, and determining the equipment type according to the parameters corresponding to the terminal equipment; determining an index data grabbing command according to the equipment type; and grabbing performance evaluation data according to the process number of the model operation command and the index data grabbing command.
In one embodiment of the application, the performance evaluation data comprises performance index display data and performance index processing data; forming a performance test report based on the performance evaluation data includes: extracting performance index processing data from the model operation log through a log data extraction instruction; processing the performance index processing data based on a preset processing rule to obtain target processing data; a performance test report is formed based on the performance index presentation data target processing data.
In one embodiment of the present application, after the performance test program is run, further comprising: and transmitting the performance index display data to a database, so that the performance index display data can be subjected to data display through grafana.
In one embodiment of the application, obtaining the model file and the test file includes: in response to the model test instructions, the model files and test files are imported from the local device by the program debug tool.
In a second aspect of the embodiments of the present application, there is provided a model test apparatus for performing the model test method as set forth in any one of the first aspects, including:
the file acquisition module is used for acquiring a model file and a test file, wherein the model file comprises an offline network model and a model execution file; the test file comprises model test data and a performance test program;
The program running module is used for running the performance test program so that the performance test program can acquire performance evaluation data based on the offline network model, the model execution file and the model test data; and
and the report generation module is used for forming a performance test report based on the performance evaluation data.
A third aspect of the present application provides an electronic apparatus, comprising: a processor; and a memory having executable code stored thereon that, when executed by the processor, causes the processor to perform the method as described above.
A fourth aspect of the application provides a non-transitory machine-readable storage medium having stored thereon executable code which, when executed by a processor of an electronic device, causes the processor to perform a method as described above.
The technical scheme provided by the embodiment of the application has the following beneficial effects:
according to the model test method, the device, the electronic equipment and the storage medium, the model file containing the offline network model and the model execution file and the test file containing the model test data and the performance test program are obtained, so that the performance test program is operated, and the performance test program can obtain the performance evaluation data based on the offline network model, the model execution file and the model test data. The method can integrate and install the offline network model, the model execution file and the model test data into the terminal equipment through the performance test program, and the performance test program can be directly applied to the subsequent offline network model and the terminal equipment in the same series after being debugged, so that complicated operations of repeatedly installing the Android Studio and configuring the Android SDK are avoided, the cost of debugging and evaluating tools is effectively reduced, and the model evaluating period, the model test efficiency and the model test threshold are reduced.
Further, the performance test report can be formed based on the performance evaluation data, and the performance evaluation data can be obtained in real time under the action of a performance test program, so that the performance test report can be synchronously formed, detailed test index data can be obtained in real time, the situation that the result hysteresis of a data curve is counted through Excel after a period of testing based on an offline SDK in the prior art is avoided, the abnormality of an offline network model can be timely found, and meanwhile, the accurate optimization requirement of the offline network model can be timely met.
In general, the technical scheme of the application can improve the testing efficiency of the offline network model, acquire detailed testing index data in real time, and meet the accurate optimization requirement of the offline network model.
Drawings
The above, as well as additional purposes, features, and advantages of exemplary embodiments of the present application will become readily apparent from the following detailed description when read in conjunction with the accompanying drawings. Several embodiments of the present application are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:
FIG. 1 schematically illustrates a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the application;
FIG. 2 schematically illustrates a flow diagram of a model test method according to an embodiment of the application;
FIG. 3 schematically shows a flow diagram of a model test method according to a further embodiment of the application;
FIG. 4 schematically shows a flow diagram of a model test method according to a further embodiment of the application;
FIG. 5 schematically shows a schematic structure of a model test apparatus according to an embodiment of the present application;
fig. 6 schematically shows a schematic block diagram of an electronic device according to an embodiment of the application.
In the drawings, the same or corresponding reference numerals indicate the same or corresponding parts.
Detailed Description
The principles and spirit of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are presented merely to enable those skilled in the art to better understand and practice the application and are not intended to limit the scope of the application in any way. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
FIG. 1 illustrates a block diagram of an exemplary computing system 100 suitable for implementing embodiments of the application. As shown in fig. 1, a computing system 100 may include: a Central Processing Unit (CPU) 101, a Random Access Memory (RAM) 102, a Read Only Memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial peripheral 112, a parallel peripheral 113, and a display 114. Of these devices, coupled to the system bus 104 are a CPU 101, a RAM 102, a ROM 103, a hard disk controller 105, a keyboard controller 106, a serial controller 107, a parallel controller 108, and a display controller 109. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that the block diagram depicted in FIG. 1 is for illustrative purposes only and is not intended to limit the scope of the present application. In some cases, some devices may be added or subtracted as the case may be.
Those skilled in the art will appreciate that embodiments of the application may be implemented as a system, method, or computer program product. Accordingly, the present disclosure may be embodied in the following forms, namely: all hardware, all software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software, is generally referred to herein as a "circuit," module, "or" system. Furthermore, in some embodiments, the application may also be embodied in the form of a computer program product in one or more computer-readable media, which contain computer-readable program code.
Any combination of one or more computer readable media may be employed. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive example) of the computer-readable storage medium could include, for example: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.
The computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.
Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
Computer program code for carrying out operations of the present application may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C ++ and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer, for example, through the internet using an internet service provider.
Embodiments of the present application will be described below with reference to flowchart illustrations of methods and block diagrams of apparatus (or systems) according to embodiments of the application. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.
According to the embodiment of the application, a model test method and device are provided.
In this document, it should be understood that any number of elements in the drawings is for illustration and not limitation, and that any naming is used only for distinction and not for any limitation.
The principles and spirit of the present application are explained in detail below with reference to several representative embodiments thereof.
Summary of The Invention
The inventor discovers that the existing test mode of the offline network model generally adopts a mode of testing based on offline SDK. The test mode needs to be used for installing the Android Studio, configuring the Android SDK and the like, but the time spent for installing the Android Studio and configuring the Android SDK is long, the operation steps are complex, the using threshold is high, and blocking problems are frequently encountered in the debugging process, such as uninitialized TTS model display and the like. Furthermore, after each iteration update of the offline network model, operations such as Android Studio installation and Android SDK configuration need to be re-executed, so that a great deal of time is required to debug tools, and under the conditions of high-frequency update iteration of the offline network model and high-frequency release of terminal equipment, the test cost is high. Moreover, by adopting the test method, performance data cannot be obtained in real time to perform model tuning, so that problems are found out later, and meanwhile, test index data corresponding to each refinement business requirement cannot be obtained, and the optimization requirement of the model cannot be met.
Based on the method, the technical scheme of the application obtains the model file containing the offline network model and the model execution file and the test file containing the model test data and the performance test program, so that the performance test program is operated, and the performance test program can obtain the performance evaluation data based on the offline network model, the model execution file and the model test data. The method can integrate and install the offline network model, the model execution file and the model test data into the terminal equipment through the performance test program, and the performance test program can be directly applied to the subsequent offline network model and the terminal equipment in the same series after being debugged, so that complicated operations of repeatedly installing the Android Studio and configuring the Android SDK are avoided, the cost of debugging and evaluating tools is effectively reduced, and the model evaluating period, the model test efficiency and the model test threshold are reduced.
Further, the performance test report can be formed based on the performance evaluation data, and the performance evaluation data can be obtained in real time under the action of a performance test program, so that the performance test report can be synchronously formed, detailed test index data can be obtained in real time, the situation that the result hysteresis of a data curve is counted through Excel after a period of testing based on an offline SDK in the prior art is avoided, the abnormality of an offline network model can be timely found, and meanwhile, the accurate optimization requirement of the offline network model can be timely met.
In general, the technical scheme of the application can improve the testing efficiency of the offline network model, acquire detailed testing index data in real time, and meet the accurate optimization requirement of the offline network model.
Having described the basic principles of the present application, various non-limiting embodiments of the application are described in detail below.
Application scene overview
The model test method of the embodiment of the application is suitable for the electronic equipment needing to apply the offline network model, and can be exemplified by a dictionary pen carrying a TTS model or an ASR model and also can be exemplified by a learning tablet equipment. Other types of electronic devices besides learning devices are also possible, and the present application is not limited in this respect, as the actual application will depend on the situation.
Exemplary method
A model test method according to an exemplary embodiment of the present application is described below with reference to fig. 2. It should be noted that the above application scenario is only shown for the convenience of understanding the spirit and principle of the present application, and the embodiments of the present application are not limited in any way. Rather, embodiments of the application may be applied to any scenario where applicable.
In step S201, a model file and a test file are acquired. In the embodiment of the present application, the model file includes an offline network model and a model execution file. The offline network model is an offline model (offline model), and is a pipe network model deployed on a client of electronic equipment such as a computer and terminal equipment and provided with a manual data updating and simulation calculation function. The offline network model may be, for example, a TTS (Text To Speech) model or an ASR (Automatic Speech Recognition, speech recognition technology) model, and it is understood that the form of the offline network model may be various, and in practical application, the actual form of the offline network model needs To be determined according To the practical application, which is not limited in this aspect of the present application.
In the embodiment of the present application, the model execution file may be a binary file, and in machine learning, the trained model generally needs to be converted into an executable form for deployment and reasoning in the production environment. This executable form is often referred to as a binary file of the model, through which the offline network model operation needs to be invoked.
In the embodiment of the present application, the test file includes model test data and a performance test program. The model test data refers to data for inputting into an offline network model for performance testing. Assuming that the offline network model is a TTS model, the model test data may be a text picture. It will be appreciated that the specific form of the model test data is various, and in practical application, the specific form of the model test data needs to be determined according to the practical application situation, and the present application is not limited in this respect.
The performance test program may be an executable file written according to a certain format, and in the embodiment of the present application, the performance test program may mainly include a python language tool and a shell command, and may be used for controlling the binary file to call the offline network model to operate, for controlling the model test data to be input into the offline network model, and for monitoring and obtaining performance index data of the offline network model. The method can integrate and install the offline network model, the model execution file and the model test data into the terminal equipment through the performance test program, and the performance test program can be directly applied to the subsequent offline network model and the terminal equipment in the same series after being debugged, so that complicated operations of repeatedly installing the Android Studio and configuring the Android SDK are avoided, the cost of debugging and evaluating tools is effectively reduced, and the model evaluating period, the model test efficiency and the model test threshold are reduced.
In step S202, a performance test program is run. After the performance test program is started to run, the performance test program can acquire performance evaluation data based on the offline network model, the model execution file and the model test data. The performance evaluation data may include data collected in each performance evaluation index, for example, data of a CPU usage index, or data of a memory occupation index, and in practical application, specific data content of the performance evaluation data needs to be determined according to practical application conditions.
In step S203, a performance test report is formed based on the performance evaluation data. In the embodiment of the application, the performance evaluation data can be further processed, and further, the performance test report can be timely summarized and formed. Therefore, detailed test index data can be obtained in real time and provided for each business party, and the business party can be a worker responsible for optimization of a model algorithm, so that the business party can timely perform performance analysis on the offline network model, timely find out the abnormality of the offline network model, improve the test efficiency, shorten the test period and simultaneously meet the accurate optimization requirement of the offline network model.
According to the technical scheme, the performance test program is operated by acquiring the model file containing the offline network model and the model execution file and the test file containing the model test data and the performance test program, so that the performance test program can acquire the performance evaluation data based on the offline network model, the model execution file and the model test data. The method can integrate and install the offline network model, the model execution file and the model test data into the terminal equipment through the performance test program, and the performance test program can be directly applied to the subsequent offline network model and the terminal equipment in the same series after being debugged, so that complicated operations of repeatedly installing the Android Studio and configuring the Android SDK are avoided, the cost of debugging and evaluating tools is effectively reduced, and the model evaluating period, the model test efficiency and the model test threshold are reduced. Further, the performance test report can be formed based on the performance evaluation data, and the performance evaluation data can be obtained in real time under the action of a performance test program, so that the performance test report can be synchronously formed, detailed test index data can be obtained in real time, the situation that the result hysteresis of a data curve is counted through Excel after a period of testing based on an offline SDK in the prior art is avoided, the abnormality of an offline network model can be timely found, and meanwhile, the accurate optimization requirement of the offline network model can be timely met. In general, the technical scheme of the application can improve the testing efficiency of the offline network model, acquire detailed testing index data in real time, and meet the accurate optimization requirement of the offline network model.
In some embodiments, the performance test program may include a test run program and a data acquisition program. The operation of the performance test program will be described in detail with reference to fig. 3. Fig. 3 schematically illustrates a flow chart of a model test method according to another embodiment of the application, referring to fig. 3, the model test method illustrated in the embodiment of the application may include:
in step S301, a test run program is run. In the embodiment of the present application, the running process of the test running program (machine_run. Sh) may specifically be: the operation parameters may be specified first, where the operation parameters include parameters corresponding to the offline network model, parameters corresponding to the model test data, parameters corresponding to the terminal device, and language parameters. The parameters corresponding to the offline network model may be a model number of the offline network model, the parameters corresponding to the model test data may be a data sample set number, the parameters corresponding to the terminal device may be a device number, and the language parameters may be a language type number.
The model run command may then be executed to cause the offline network model to initiate operation.
And then, the running starting time of the offline network model can be recorded, when the running time of the offline network model reaches the preset time, the running process of the offline network model is closed, and the running ending time of the offline network model is recorded.
Finally, after the performance evaluation data is captured, a model operation log (log) can be formed based on the performance evaluation data, the operation starting time, the operation ending time and the operation parameters.
In step S302, after executing the model run command in the test run program, the model execution file is enabled to call the offline network model for running in response to the model run command. Because the offline network model cannot be invoked, a model execution file needs to be enabled to invoke the offline network model for operation. It will be appreciated that after the offline network model is run, the model test data may be entered into the offline network model for processing, and during this processing, the performance evaluation data of the offline network model may need to be monitored and captured.
In step S303, the data acquisition program is run in response to the model running command in the test running program, and the performance evaluation data is obtained. In the embodiment of the present application, the running process of the data acquisition program (machine_catch. Sh) may specifically be: the device type may first be determined in response to a model run command, based on parameters corresponding to the terminal device. In embodiments of the application, the device types may include, but are not limited to, a dictionary pen device, a learning tablet device, and a learning desk lamp device. It will be appreciated that the device type may be varied, and in practical applications, the content included in the device type needs to be determined according to the practical application, and the present application is not limited in this respect.
The index data grab command may then be determined based on the device type. Since different device types may have different run codes, different index data grab commands may be required for different device types. Although different equipment types may require different index data capture commands, the same performance profile may ultimately be captured. In an embodiment of the present application, the index data grabbing command may be a regular expression, and illustratively, the regular expression for the dictionary pen device may be mem= $ { result/K/"}. It will be appreciated that the form of the index data capture command may be varied, and in practical applications, the specific form of the index data capture command needs to be determined according to practical application conditions, and the present application is not limited in this respect.
And finally, grabbing performance evaluation data according to the process number of the model operation command and the index data grabbing command. In the embodiment of the application, the process corresponding to the model operation command can be screened and inquired in the operation process, so that the process number corresponding to the model operation command can be determined. And then the performance evaluation data under the process number corresponding to the model operation command can be grabbed through the index data grabbing command.
In some embodiments, the performance profile may include, but is not limited to, performance index presentation data and performance index processing data, wherein the performance index presentation data can be used for data presentation. In addition, the performance index processing data needs to be further processed, and a performance test report is formed together with the performance index display data after the processing. Meanwhile, the acquisition process of the model file and the test file can be further designed. The process of forming the performance test report and the process of obtaining the model file and the test file will be described in detail with reference to fig. 4. Fig. 4 schematically illustrates a flow chart of a model test method according to still another embodiment of the present application, referring to fig. 4, the model test method illustrated in the embodiment of the present application may include:
in step S401, in response to the model test instruction, a model file and a test file are imported from the local device by the program debugging tool. The model test instruction may be formed by active input of a user of the terminal device, or may be formed by autonomous formation of the terminal device to perform self-test after the offline network model is run for a certain time. It will be appreciated that the generation mode of the model test instruction may be various, and in practical application, the generation mode of the model test instruction needs to be determined according to the practical application situation, and the application is not limited in this respect.
The program debugging tool can be Android Debug Bridge (adb), is a multifunctional instruction sequence tool, and can realize the communication between the local equipment and the terminal equipment. The local device may refer to an electronic device such as a computer that is deployed locally. The local device may be instructed to download the model file from the server prior to importing the model file and the test file from the local device via the program debug tool, where the download instruction may be illustratively "tsh scp xxx".
It should be understood that the foregoing description of the obtaining manners of the model file and the test file is merely exemplary, and in practical application, the obtaining manners of the model file and the test file need to be determined according to practical application conditions, and the present application is not limited in this respect.
In step S402, a performance test program is run. After the performance test program is started to run, the performance test program can acquire performance evaluation data based on the offline network model, the model execution file and the model test data.
In an embodiment of the present application, the performance evaluation data may include, but is not limited to, performance index display data and performance index processing data. The performance index presentation data may include, but is not limited to, CPU resource utilization (single core average), CPU resource utilization (single core MAX), TPU memory occupancy (average), CPU physical memory occupancy (MAX), CPU physical memory occupancy (average), CPU physical memory occupancy (MAX), and the like. Additionally, the performance index processing data may include, but is not limited to, TPU initialization occupancy, average initialization time, average real-time rate, P99 real-time rate, average latency, P99 latency, average handover language initialization time, P99 handover language initialization time, and the like. Wherein, the above-mentioned "average" means to average the corresponding index data between the running start time and the running end time of the offline network model; the above-mentioned "MAX" refers to performing maximum value processing on the corresponding index data between the operation start time and the operation end time of the offline network model; the term "P99" refers to a process of calculating the number of bits of the corresponding index data between the operation start time and the operation end time of the offline network model, and the calculating process refers to a process of calculating the number points in which the probability distribution range of one random variable is divided into 99 equal parts.
In step S403, the performance index display data is transmitted to the database, so that the performance index display data can be displayed by grafana. The grafana is a data visualization platform and can meet the data query, visualization and alarm requirements of different scenes such as operation and maintenance monitoring, operation analysis, report display and the like.
In step S404, a performance test report is formed based on the performance evaluation data. Specifically, the performance index processing data can be extracted from the model operation log (log) through a log data extraction instruction. The log data extraction instructions may illustratively be regular expressions that are capable of extracting all data of all performance metrics from a model log of operations (log) that require further processing.
And then processing the performance index processing data based on a preset processing rule to obtain target processing data. The preset processing rule may be "average" and/or "P99" described in step S402. It will be appreciated that the specific rule content of the preset processing rule may be various, and in practical application, the specific rule content of the preset processing rule needs to be determined according to the practical application situation, and the present application is not limited in this aspect.
And finally, summarizing and sorting the target processing data based on the performance index display data to form a performance test report. Therefore, the service party can timely compare the data in the performance test report with the expected target data, and further timely determine the performance stability of the offline network model. If the abnormality is found, the offline network model can be accurately optimized according to the detailed data in the performance test report.
Exemplary apparatus
Having introduced the method of the exemplary embodiment of the present application, next, a description will be given of the related products of the model test method of the exemplary embodiment of the present application with reference to fig. 5 and 6.
Fig. 5 schematically shows a schematic structural view of a model test apparatus according to an embodiment of the present application. Referring to fig. 5, a model test apparatus according to an embodiment of the present application may include:
the file acquisition module 501 is configured to acquire a model file and a test file, where the model file includes an offline network model and a model execution file; the test file comprises model test data and a performance test program;
the program running module 502 is configured to run a performance test program, so that the performance test program can obtain performance evaluation data based on an offline network model, a model execution file and model test data; and
A report generation module 503 for forming a performance test report based on the performance evaluation data.
According to the model test device, the model files containing the offline network model and the model execution file and the test files containing the model test data and the performance test program are obtained, so that the performance test program is operated, and the performance evaluation data can be obtained based on the offline network model, the model execution file and the model test data. The method can integrate and install the offline network model, the model execution file and the model test data into the terminal equipment through the performance test program, and the performance test program can be directly applied to the subsequent offline network model and the terminal equipment in the same series after being debugged, so that complicated operations of repeatedly installing the Android Studio and configuring the Android SDK are avoided, the cost of debugging and evaluating tools is effectively reduced, and the model evaluating period, the model test efficiency and the model test threshold are reduced.
Further, the performance test report can be formed based on the performance evaluation data, and the performance evaluation data can be obtained in real time under the action of a performance test program, so that the performance test report can be synchronously formed, detailed test index data can be obtained in real time, the situation that the result hysteresis of a data curve is counted through Excel after a period of testing based on an offline SDK in the prior art is avoided, the abnormality of an offline network model can be timely found, and meanwhile, the accurate optimization requirement of the offline network model can be timely met.
In general, the technical scheme of the application can improve the testing efficiency of the offline network model, acquire detailed testing index data in real time, and meet the accurate optimization requirement of the offline network model.
Fig. 6 schematically shows a schematic block diagram of an electronic device according to an embodiment of the application. Referring to fig. 6, an electronic device 600 may include a processor 601. Further, the electronic device may also comprise a memory 602 storing computer instructions that, when executed by the processor 601, cause the electronic device 600 to perform the method according to the various embodiments or implementations described above.
In some implementations, the electronic device 600 may include a server or a terminal device, such as a physical server, a cloud server, a server cluster, a data processing apparatus, an application testing robot, a computer terminal, a smart terminal, a PC device, an internet of things terminal, and so forth.
Depending on the implementation scenario, the above processor 601 may be a central processing unit (Central Processing Unit, CPU), but may also be other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Based on the foregoing, the present application also discloses a computer readable storage medium containing program instructions which, when executed by a processor, cause the implementation of the method according to the various embodiments or implementations described above.
In some implementations, the computer-readable storage medium may be any suitable magnetic or magneto-optical storage medium, such as, for example, resistive Random Access Memory RRAM (Resistive Random Access Memory), dynamic Random Access Memory DRAM (Dynamic Random Access Memory), static Random Access Memory SRAM (Static Random-Access Memory), enhanced dynamic Random Access Memory EDRAM (Enhanced Dynamic Random Access Memory), high-Bandwidth Memory HBM (High-Bandwidth Memory), hybrid storage cube HMC (Hybrid Memory Cube), etc., or any other medium that may be used to store the desired information and that may be accessed by an application, a module, or both. Any such computer storage media may be part of, or accessible by, or connectable to, the device. Any of the applications or modules described herein may be implemented using computer-readable/executable instructions that may be stored or otherwise maintained by such computer-readable media.
It should be noted that although several devices or sub-devices of the model test device are mentioned in the above detailed description, this division is not mandatory only. Indeed, the features and functions of two or more of the devices described above may be embodied in one device, in accordance with embodiments of the present application. Conversely, the features and functions of one device described above may be further divided into multiple devices to be embodied.
Furthermore, although the operations of the methods of the present application are depicted in the drawings in a particular order, this is not required to either imply that the operations must be performed in that particular order or that all of the illustrated operations be performed to achieve desirable results. Rather, the steps depicted in the flowcharts may change the order of execution. Additionally or alternatively, certain steps may be omitted, multiple steps combined into one step to perform, and/or one step decomposed into multiple steps to perform.
Use of the verb "comprise," "include" and its conjugations in this application does not exclude the presence of elements or steps other than those stated in the application. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements.
While the spirit and principles of the present application have been described with reference to several particular embodiments, it is to be understood that the application is not limited to the disclosed embodiments nor does it imply that features of the various aspects are not useful in combination, nor are they useful in any combination, such as for convenience of description. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims (10)

1. A method of model testing, comprising:
obtaining a model file and a test file, wherein the model file comprises an offline network model and a model execution file; the test file comprises model test data and a performance test program;
running the performance test program so that the performance test program can acquire performance evaluation data based on the offline network model, the model execution file and the model test data; and
and forming a performance test report based on the performance evaluation data.
2. The model test method according to claim 1, wherein the performance test program includes a test run program and a data acquisition program;
The running the performance test program includes:
running the test running program; and
and responding to a model operation command in the test operation program to operate the data acquisition program to obtain the performance evaluation data.
3. The model test method of claim 2, wherein the running the test run program comprises:
specifying operation parameters, wherein the operation parameters comprise parameters corresponding to the offline network model, parameters corresponding to the model test data, parameters corresponding to terminal equipment and language parameters;
executing the model operation command to enable the offline network model to start operation;
recording the running starting time of the offline network model;
when the operation time length of the offline network model reaches a preset time length, closing the operation process of the offline network model, and recording the operation ending time of the offline network model; and
and forming a model operation log based on the performance evaluation data, the operation starting time, the operation ending time and the operation parameters.
4. A model test method according to claim 3, further comprising, after said executing said model run command:
And enabling the model execution file to call the offline network model to operate in response to the model operation command.
5. A model test method according to claim 3, wherein the running the data acquisition program in response to a model running command in the test running program comprises:
responding to the model operation command, and determining the equipment type according to the parameters corresponding to the terminal equipment;
determining an index data grabbing command according to the equipment type; and
and grabbing the performance evaluation data according to the process number of the model operation command and the index data grabbing command.
6. A model test method according to claim 3, wherein the performance evaluation data comprises performance index display data and performance index processing data;
the forming a performance test report based on the performance evaluation data comprises:
extracting the performance index processing data from the model operation log through a log data extraction instruction;
processing the performance index processing data based on a preset processing rule to obtain target processing data;
the target process data forms the performance test report based on the performance index display data.
7. The model test method of claim 6, further comprising, after said running said performance test program:
and transmitting the performance index display data to a database, so that the performance index display data can be subjected to data display through grafana.
8. Model testing apparatus for performing the model testing method according to any one of claims 1-7, comprising:
the file acquisition module is used for acquiring a model file and a test file, wherein the model file comprises an offline network model and a model execution file; the test file comprises model test data and a performance test program;
the program running module is used for running the performance test program so that the performance test program can acquire performance evaluation data based on the offline network model, the model execution file and the model test data; and
and the report generation module is used for forming a performance test report based on the performance evaluation data.
9. An electronic device, comprising:
a processor; and
a memory having stored thereon executable code for model testing, which when executed by the processor, causes the processor to perform the method of any of claims 1-7.
10. A non-transitory machine-readable storage medium having stored thereon executable code for model testing, which when executed by a processor of an electronic device, causes the processor to perform the method of any of claims 1-7.
CN202310840748.1A 2023-07-10 2023-07-10 Model test method, device, electronic equipment and storage medium Pending CN116775438A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310840748.1A CN116775438A (en) 2023-07-10 2023-07-10 Model test method, device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310840748.1A CN116775438A (en) 2023-07-10 2023-07-10 Model test method, device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN116775438A true CN116775438A (en) 2023-09-19

Family

ID=87989491

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310840748.1A Pending CN116775438A (en) 2023-07-10 2023-07-10 Model test method, device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN116775438A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117933428A (en) * 2024-03-20 2024-04-26 清华大学 Method, device and equipment for evaluating data offset coping strategy

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117933428A (en) * 2024-03-20 2024-04-26 清华大学 Method, device and equipment for evaluating data offset coping strategy

Similar Documents

Publication Publication Date Title
US11321122B2 (en) Method, apparatus, device and medium for processing topological relation of tasks
CN110852438B (en) Model generation method and device
WO2020087974A1 (en) Model generation method and device
CN110807515A (en) Model generation method and device
US20240348628A1 (en) Cybersecurity investigation tools utilizing information graphs
CN109743311B (en) WebShell detection method, device and storage medium
CN115427968A (en) Robust artificial intelligence reasoning in edge computing devices
CN111340220B (en) Method and apparatus for training predictive models
CN111523640A (en) Training method and device of neural network model
CN114580263A (en) Knowledge graph-based information system fault prediction method and related equipment
CN110515944B (en) Data storage method based on distributed database, storage medium and electronic equipment
CN111353601B (en) Method and apparatus for predicting latency of model structure
US20230115163A1 (en) Method for processing data, and electronic device, storage medium and program product
US20190272471A1 (en) System and method for deploying a data analytics model in a target environment
CN112783508B (en) File compiling method, device, equipment and storage medium
CN116701031A (en) Root cause model training method, analysis method and device in micro-service system
CN116775438A (en) Model test method, device, electronic equipment and storage medium
WO2019104304A1 (en) Accelerated simulation setup process using prior knowledge extraction for problem matching
WO2015003452A1 (en) Methods and systems for file processing
CN115905450A (en) Unmanned aerial vehicle monitoring-based water quality abnormity tracing method and system
US20150020056A1 (en) Methods and systems for file processing
CN113434123A (en) Service processing method and device and electronic equipment
US20230315613A1 (en) Method and system for eradicating programmatical errors from engineering programs in a technical installation
CN111047427A (en) Data reporting method, device, server and storage medium
CN115834195A (en) Log anomaly detection method, device, system and medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination