CN114694007A

CN114694007A - Video structuring processing method and video structuring engine

Info

Publication number: CN114694007A
Application number: CN202011568421.6A
Authority: CN
Inventors: 程冰; 宁哲
Original assignee: Shenzhen Intellifusion Technologies Co Ltd
Current assignee: Shenzhen Intellifusion Technologies Co Ltd
Priority date: 2020-12-25
Filing date: 2020-12-25
Publication date: 2022-07-01

Abstract

The application is applicable to the technical field of video data processing, and provides a video structured processing method and a video structured engine, which comprise the following steps: when a video structuring processing request is received, preprocessing the video structuring processing request to acquire resolution information of video data to be processed in the video structuring processing request; calculating a video memory value consumed by processing the video data to be processed according to the resolution information of the video data to be processed, a preset resolution reference value and a preset distribution video memory value; if the video memory value consumed for processing the video data to be processed is less than or equal to the residual video memory resource, performing video structuring processing on the video data to be processed; and if the video memory value consumed for processing the video data to be processed is larger than the residual video memory resource, stopping responding to the video structured processing request, and judging whether the video memory value consumed for processing the video data to be processed is larger than the residual video memory resource again after a preset time interval, thereby effectively avoiding the situation of video memory overflow.

Description

Video structuring processing method and video structuring engine

Technical Field

The present application belongs to the technical field of video data processing, and in particular, to a video structured processing method, a video structured engine, a terminal device, and a computer readable storage medium.

Background

The video structuralization processing refers to a processing process of intelligently analyzing people, vehicles, objects, colors and the like in a video picture, extracting key information and performing semantic description on a text. The video structured big data application is widely applied to scenes such as city security, smart communities, new commercial systems and the like. In the video structuring process, it is usually necessary to decode video data through a function provided by a hardware acceleration device, and then perform structuring processing on the decoded video data through a video structuring engine.

However, video data to be processed in a video structure may come from multiple manufacturers and various video source equipment, and has the characteristic of diverse resolution. The video structuring engine generally configures hardware acceleration equipment of different specifications aiming at different server resources or different periods of market demands, and video memory resources in the hardware acceleration equipment of different specifications are different in size, so that the number of video paths that the hardware acceleration equipment of different specifications can analyze is also different, and the resolution of accessed video data is not necessarily the same, so that the situation of video memory overflow of the hardware acceleration equipment is easy to occur, the stability of the video structuring engine is influenced, the whole video structuring system is broken down, and the value loss which cannot be evaluated is caused.

Disclosure of Invention

In view of this, the present application provides a video structuring processing method, a video structuring engine, a terminal device, and a computer readable storage medium, so as to solve the problem that in the existing video structuring processing process, in the processing process for a multi-resolution video source, the video memory of a hardware acceleration device is easy to overflow, which further affects the stability of the video structuring engine, thereby causing the breakdown of the whole video structuring system.

A first aspect of an embodiment of the present application provides a video structured processing method, including:

when a video structuring processing request is received, preprocessing the video structuring processing request to acquire resolution information of video data to be processed in the video structuring processing request;

calculating a video memory value consumed by processing the video data to be processed according to the resolution information of the video data to be processed, a preset resolution reference value and a preset distribution video memory value;

if the video memory value consumed for processing the video data to be processed is less than or equal to the residual video memory resource, performing video structuring processing on the video data to be processed;

and if the video memory value required to be consumed for processing the video data to be processed is larger than the residual video memory resource, stopping responding to the video structuralized processing request, and judging whether the video memory value required to be consumed for processing the video data to be processed is larger than the residual video memory resource again after a preset time interval.

Illustratively, a specific calculation formula for calculating the video memory value that needs to be consumed by the hardware acceleration device when processing the video data to be processed according to the resolution information of the video data to be processed, the preset resolution reference value, and the preset allocated video memory value is as follows:

M_use＝R÷K×A；

wherein M is_useThe video memory value consumed for processing the video data to be processed is referred to, R is the resolution of the video data to be processed, K is a preset resolution reference value, and A is a preset allocated video memory value.

Further, the video structuring processing method further includes:

an initialization operation is performed.

Further, the performing an initialization operation includes:

setting the maximum occupied video memory resource according to the size of the video card resource of the hardware acceleration equipment;

setting the residual video memory resources according to the video memory resources occupied to the maximum extent;

acquiring resolution information of each video source device;

setting a preset resolution reference value according to the resolution information of each video source device;

and setting a preset distribution video memory value according to the preset resolution reference value.

Further, the food structuring processing method further comprises the following steps:

and updating the residual video memory resources in real time according to the video structuring processing progress.

Further, the updating the remaining video memory resources in real time according to the video structuring processing progress includes:

when the video structuring engine carries out structuring processing on the video data to be processed, the residual video memory resources are updated to the original residual video memory resources minus the video memory value consumed when the video data to be processed is processed.

Further, the updating the remaining video memory resources in real time according to the video structuring processing progress further includes:

and after the video structuring engine finishes the data structuring processing, updating the residual video memory resources into the original residual video memory resources and the video memory value consumed when the video data to be processed is processed.

Further, if the video memory value consumed for processing the video data to be processed is less than or equal to the remaining video memory resources, performing video structuring processing on the video data to be processed includes:

decoding a video file in the video structuring processing request to obtain video data to be processed;

and carrying out structuring processing on the video data to be processed.

A second aspect of an embodiment of the present application provides a video structuring engine, including:

the device comprises a preprocessing unit, a processing unit and a processing unit, wherein the preprocessing unit is used for preprocessing a video structuring processing request when the video structuring processing request is received, and acquiring resolution information of video data to be processed in the video structuring processing request;

the calculation unit is used for calculating the video memory value consumed by processing the video data to be processed according to the resolution information of the video data to be processed, a preset resolution reference value and a preset allocated video memory value;

the structuralization processing unit is used for carrying out video structuralization processing on the video data to be processed if the video memory value consumed for processing the video data to be processed is less than or equal to the residual video memory resource;

and the pause unit is used for pausing the response to the video structured processing request if the video memory value which needs to be consumed for processing the video data to be processed is larger than the residual video memory resource, and judging whether the video memory value which needs to be consumed for processing the video data to be processed is larger than the residual video memory resource again after a preset time interval.

A third aspect of the embodiments of the present application provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the terminal device, where the processor implements the steps of the video structured processing method provided in the first aspect when executing the computer program.

A fourth aspect of embodiments of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the video structured processing method provided in the first aspect.

A fifth aspect of embodiments of the present application provides a computer program product, which, when run on a terminal device, causes the terminal device to execute the steps of the video structuring processing method according to any one of the above first aspects.

The video structuring processing method, the video structuring engine, the terminal device and the computer readable storage medium provided by the embodiment of the application have the following beneficial effects that:

according to the video structuring processing method, the resolution information of the video data to be processed is obtained, the video memory resources which need to be consumed when the video data are processed are determined based on the resolution information of the video data to be processed, and the video data to be processed are processed under the condition that the residual video memory resources are sufficient, so that the video structuring concurrent path number of a multi-resolution video source is adaptively controlled, the condition of video memory overflow is effectively avoided, and the stability of the whole video structuring system during multi-resolution concurrent structuring is maintained.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive exercise.

Fig. 1 is a flowchart of an implementation of a video structuring processing method provided in an embodiment of the present application;

fig. 2 is a schematic structural diagram of a video structuring engine adapted to a video structuring processing method according to an embodiment of the present application

Fig. 3 is a flowchart of an implementation of a video structuring processing method according to another embodiment of the present application;

fig. 4 is a schematic structural diagram of a video structuring engine provided in an embodiment of the present application;

fig. 5 is a schematic structural diagram of a terminal device according to another embodiment of the present application.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

As used in this specification and the appended claims, the term "if" may be interpreted contextually as "when", "upon" or "in response to a determination" or "in response to a detection". Similarly, the phrase "if it is determined" or "if a [ described condition or event ] is detected" may be interpreted contextually to mean "upon determining" or "in response to determining" or "upon detecting [ described condition or event ]" or "in response to detecting [ described condition or event ]".

Furthermore, in the description of the present application and the appended claims, the terms "first," "second," "third," and the like are used for distinguishing between descriptions and not necessarily for describing a relative importance or importance.

Reference throughout this specification to "one embodiment" or "some embodiments," or the like, means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," or the like, in various places throughout this specification are not necessarily all referring to the same embodiment, but rather "one or more but not all embodiments" unless specifically stated otherwise. The terms "comprising," "including," "having," and variations thereof mean "including, but not limited to," unless expressly specified otherwise.

Referring to fig. 1, fig. 1 is a flowchart illustrating an implementation of a video structuring processing method according to an embodiment of the present disclosure. In this embodiment, the video structuring processing method is used to control the number of video structuring concurrent paths when performing the structuring processing on the video data, so as to avoid the occurrence of video memory overflow, and effectively maintain the stability of the entire video structuring system during the multi-resolution concurrent structuring, and the execution subject of the video structuring processing method may be a computer terminal, a server, a cloud server, or the like having a video structuring processing function.

In the embodiment of the application, the video structuring processing method is mainly applied to a video structuring big data cloud platform. The video structured engine of the video structured big data cloud platform can perform structured processing on video data, meanwhile, decoding of the video data can be achieved through a function provided by hardware acceleration equipment, and the decoded video data can be subjected to structured processing through the video structured engine.

In practical applications, the existing video structuring engine may use hardware acceleration devices of different specifications to assist video decoding for different server resources and market plans, and the video memory resources of hardware acceleration devices of different specifications are different, so that the number of video paths that can be concurrently processed by each hardware acceleration device is also different. Moreover, the video data accessed by the video structured big data cloud platform has the characteristic of diversity of resolution, so that certain hardware acceleration equipment can be stably processed when continuously and concurrently processing N paths of data with the same resolution, but when the video data accessed by a certain path is changed into video data with higher resolution, the video memory resources of the hardware acceleration equipment are insufficient, and the analysis of the video data is difficult to support, so that the function library of the hardware acceleration equipment is output once, namely, the video memory overflows. This affects the stability of the video structuring engine, resulting in a breakdown of the entire video structuring system, with an immeasurable loss of value.

In order to solve the above problems, according to the video structured processing method provided by the application, resolution information of video data to be processed is obtained, video memory resources required to be consumed when the video data is processed are determined based on the resolution information of the video data to be processed, and the video data to be processed is processed only when the remaining video memory resources are sufficient, so that adaptive control over the number of video structured concurrent paths of a multi-resolution video source is realized, the situation of video memory overflow is effectively avoided, and the stability of the whole video structured system during multi-resolution concurrent structuring is maintained.

Referring to fig. 2, fig. 2 is a schematic structural diagram illustrating a video structuring engine to which the video structuring processing method according to the embodiment of the present disclosure is applied. As shown in fig. 2, the video structuring engine 20 may be connected to a plurality of video source devices and may be externally connected to a plurality of hardware acceleration devices. The video structuring engine 20 may include a video preprocessing module 21, a video structuring decision module 22, a video concurrent path number resource pool management module 23, and a video structuring processing module 24.

The video preprocessing module 21 may be configured to, when receiving a video structural processing request, obtain a video file or a video stream file included in the video structural processing request, then package and analyze the video file or the video stream file, and then obtain resolution information corresponding to video data to be processed in the video structural processing request.

The video structuring decision module 22 may be configured to calculate a video memory value that needs to be consumed for processing the video data to be processed, according to the resolution information of the video data to be processed, which is acquired by the video preprocessing module 21, and a preset allocated video memory value that is memorized with a preset resolution reference value. And then, deciding whether to continue responding to the video structured processing request or not based on the difference value between the video memory value required to be consumed for processing the video data to be processed and the residual video memory resource.

The video concurrent path number resource pool management module 23 is configured to obtain the remaining video memory resource of the hardware acceleration device connected to the video structuring engine.

The video preprocessing module 21 may also be configured to decode a video file or a video stream file in the video structured processing request when the video structured decision module 22 decides that video structured processing needs to be performed on the video data to be processed, so as to obtain the video data to be processed.

The video structuring processing module 24 is configured to perform the structuring processing on the video data to be processed when the video structuring decision module 22 decides that the video structuring processing needs to be performed on the video data to be processed.

It should be noted that the video source device may be used to collect video data, and may also be used to store video data. The hardware acceleration device described above may be used to decode video data. For example, the video source device may be a terminal device such as a video camera having a video capture function, and the hardware acceleration device may be a device such as an operation acceleration card capable of providing GPU resources.

Referring to fig. 1, fig. 1 is a schematic diagram illustrating an implementation flow of a video structuring processing method according to an embodiment of the present application, and as shown in fig. 1, the video structuring processing method includes the following steps:

s11: when a video structuring processing request is received, preprocessing the video structuring processing request to acquire resolution information of video data to be processed.

In this embodiment of the present application, the video source device may initiate the video structuring engine, and the video structuring engine, when receiving the video structuring request, performs preprocessing on the video structuring request, so as to obtain resolution information of to-be-processed video data requested to be processed in the video structuring request.

It should be noted that the preprocessing of the video structuring request refers to performing encapsulation and parsing on the received video structuring request.

It should be noted that the video source device may be a video capture terminal, or may be a storage terminal that stores a video file, which is not limited herein. In addition, the video structuring processing request may also be initiated by other devices, for example, by a user equipment, which is not limited in this application.

In the embodiment of the present application, the resolution information refers to a resolution size of data to be processed in a video file/video stream file included in the video structuring processing request.

It should be noted that the resolution of the video data acquired by each video source device is related to the acquisition parameters of the video source device. When the video source device collects video data and initiates a video structuring processing request, the related resolution information is written into the video structuring processing request. After receiving the video structuring processing request, the video structuring engine performs encapsulation analysis processing on the video structuring processing request to obtain the resolution information.

Here, after the video source device acquires the video data, the video data is encapsulated based on a network transmission protocol between the video source device and the video structuring engine, and the video data and related information (including resolution information) of the video data are encapsulated into a video structuring processing request conforming to the network transmission protocol. The encapsulation parsing refers to an operation of parsing a video structured processing request after video data is encapsulated based on a network transport protocol.

It should be further noted that, for performing encapsulation analysis on the video structuring processing request, reference may be made to an existing analysis method, which is not described herein again in this embodiment of the present application.

S12: and calculating the video memory value consumed by processing the video data to be processed according to the resolution information of the video data to be processed, a preset resolution reference value and a preset distribution video memory value.

In the embodiment of the present application, the preset resolution reference value may be set according to an actual application scenario.

Specifically, the preset resolution reference value may be set as a resolution value that is the lowest resolution parameter of video source equipment connected to the video structuring engine and is used as the preset resolution reference value. For example, if there are 100 w/in video source device a, 200 w/in video source device B, and 500 w/in video source device C among the video source devices connected to the video structuring engine, the video source device with the lowest resolution parameter is video source device a, and thus the preset resolution reference value is set to 100 w/in. It is understood that the preset resolution reference value may also be set according to other setting criteria, for example, the preset resolution reference value is directly set to a certain fixed value, for example, the preset resolution reference value is directly set to 200w pixels/inch. It should be noted that the above is only an example of setting the preset resolution reference value and is not limited.

In this embodiment, the preset allocated video memory value corresponds to the preset resolution reference value. The preset allocated video memory value may be set as a value of a video memory resource that needs to be consumed when the hardware acceleration device processes the video data at the resolution (preset resolution reference value) through the provided function library.

It should be noted that, after determining the resolution of the video data, the video structuring engine can know how much video memory resources will be occupied after the video data is decoded, and the hardware acceleration device will also occupy a part of the video memory resources when calling the function of the hardware acceleration device to perform decoding processing, so that when setting the preset allocated video memory value, not only how much video memory resources will be occupied after decoding, but also the video memory resources that need to be consumed during decoding need to be considered. The video memory resources required to be consumed during decoding are uncontrollable, so that the average value of the video memory resources required to be consumed for processing the video data with a certain resolution during decoding can be determined through multiple tests, and a certain floating interval is given to ensure that the video memory resources are sufficient.

In an embodiment of the present application, a specific calculation formula for calculating a video memory value that needs to be consumed by the hardware acceleration device when processing the video data to be processed according to the resolution information of the video data to be processed, the preset resolution reference value, and the preset allocated video memory value is as follows:

M_use＝R÷K×A；

S13: and if the video memory value consumed for processing the video data to be processed is less than or equal to the residual video memory resource, performing video structuring processing on the video data to be processed.

In the embodiment of the application, after the video memory value required to be consumed for processing the video data to be processed is determined, the remaining video memory resource of the hardware acceleration device can be obtained in real time. The residual video memory resource of the hardware technology equipment can be obtained by the video concurrent path number resource pool management module.

Specifically, each hardware acceleration device provides a software interface for acquiring the video memory resource of the hardware acceleration device, and when no video data is processed at first, the hardware acceleration device may be controlled to initialize, and the video memory resource of the hardware acceleration device is acquired through the software interface, where the acquired video memory resource is the maximum video memory resource that can be used by the hardware acceleration device.

In this embodiment of the application, after the remaining video memory resource is obtained, if a video memory value consumed for processing the to-be-processed data in the video structuring processing request is less than or equal to the remaining video memory resource, it indicates that the external hardware acceleration device has sufficient video memory resources to process the video data. Therefore, the video structuring processing can be further carried out on the video data to be processed.

In an embodiment of the present application, the process of performing video structuring processing on the video data to be processed may include the following steps:

and carrying out structural processing on the video data to be processed.

In a specific application, decoding the video file in the video structuring processing request may be implemented by a function provided by an external hardware acceleration device.

In a specific application, the function functions of the hardware acceleration device include, but are not limited to: interface function and implementation function, such as abstract interface function of chroma migration of WebKit, and implementation function of WebLayer.

It should be noted that the structural processing of the video data refers to a process of performing image analysis and key feature extraction on people or objects (including vehicles and the like) in the video data, and describes features of the people or objects through text semantics. For example, when people in the video data need to be structured, the gender feature, the skin color feature, the clothing feature, the posture feature, etc. of people in each frame of image in the video data can be extracted, and then the above features are described through words.

S14: and if the video memory value required to be consumed for processing the video data to be processed is larger than the residual video memory resource, stopping responding to the video structuralized processing request, and judging whether the video memory value required to be consumed for processing the video data to be processed is larger than the residual video memory resource again after a preset time interval.

In this embodiment of the present application, if the video memory value consumed for processing the to-be-processed data in the video structured processing request is greater than the remaining video memory resource, it indicates that the video memory resource of the external hardware acceleration device cannot meet the requirement for processing the video data. If the video structuring processing request is continuously responded, the situation of video memory overflow can be caused, so that the video structuring processing request can only be temporarily responded, and whether the video memory value consumed for processing the video data to be processed is larger than the residual video memory resource or not is judged again after a preset time interval is waited. And if the video memory value consumed for processing the video data to be processed is still larger than the residual video memory resource after waiting for the preset time interval, continuing waiting. And if the video memory value consumed for processing the video data to be processed is less than or equal to the residual video memory resource after waiting for the preset time interval, indicating that the external hardware acceleration equipment has sufficient video memory resources to process the video data. Therefore, the video structuring processing can be further carried out on the video data to be processed.

It should be noted that the preset time interval may be set according to actual requirements, for example, set to 3 seconds.

As can be seen from the above, in the video structured processing method provided in this embodiment, resolution information of video data to be processed is obtained, video memory resources that need to be consumed when the video data is processed are determined based on the resolution information of the video data to be processed, and the video data to be processed is processed only when the remaining video memory resources are sufficient, so that adaptive control over the number of video structured concurrent paths of a multi-resolution video source is achieved, the situation of video memory overflow is effectively avoided, and the stability of the whole video structured system during multi-resolution concurrent structuring is maintained.

Referring to fig. 3, fig. 3 is a schematic view illustrating an implementation flow of a video structuring processing method according to another embodiment of the present application. As shown in fig. 3, different from the previous embodiment, the video structuring processing method provided in the embodiment of the present application further includes the following steps:

s31: an initialization operation is performed.

In the embodiment of the present application, before the video structuring engine does not start the video structuring process, an initialization operation may be performed.

Specifically, the initialization operation may include operations of acquiring resolution information of each video source device, setting a preset resolution reference value according to the resolution information of each video source device, setting a preset allocated video memory value according to the preset resolution reference value, and the like.

In an embodiment of the present application, the initialization operation may further include the following steps:

and setting the residual video memory resources according to the maximum occupied video memory resources.

Specifically, during initialization operation, the video structuring engine may first obtain the size of the graphics card resource of the hardware acceleration device to set the maximum occupied graphics memory resource. The maximum occupied video memory resource is the maximum video memory resource that the hardware acceleration device can provide.

When the initialization is started, since the hardware acceleration device does not perform the video decoding processing yet, the initialized remaining video memory resource can be set to occupy the video memory resource maximally. It can be understood that the remaining video memory resources may also be set to a certain value within the video memory resources occupied to the maximum extent, which is not limited in the present application.

S32: and updating the residual video memory resources in real time according to the video structuring processing progress.

In the embodiment of the application, when the video structuring engine determines that the video data to be processed needs to be structured, the remaining video memory resource is updated to the original remaining video memory resource minus the video memory value occupied (consumed) for processing the video data to be processed.

In the embodiment of the present application, after the video structuring engine completes the data structuring process, the occupied video memory resource is released to the remaining video memory resource pool, that is, the remaining video memory resource is updated to the original remaining video memory resource plus the video memory value occupied (consumed) for processing the video data to be processed.

As can be seen from the above, in the video structuring processing method provided in the embodiment of the present application, each parameter is set through initialization operation, so as to determine resolution information for processing video data to be processed, and the remaining video memory resources can be updated in real time, so that the video structuring engine can perform adaptive control on the number of video structured concurrent channels according to the remaining video memory resources updated in real time, thereby effectively avoiding the situation of video memory overflow, and maintaining the stability of the entire video structuring system during multi-resolution concurrent structuring.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

Referring to fig. 4, fig. 4 is a block diagram illustrating a video structuring engine according to an embodiment of the present disclosure. The video structuring engine in this embodiment comprises units for performing the steps in the corresponding embodiment of fig. 1. Please refer to fig. 1 and related descriptions in the embodiment corresponding to fig. 1. For convenience of explanation, only the portions related to the present embodiment are shown. Referring to fig. 4, the video structuring engine 40 includes: a preprocessing unit 41, a calculation unit 42, a structuring processing unit 43 and a pause unit 44. Wherein:

the preprocessing unit 41 is configured to, when receiving a video structuring processing request, preprocess the video structuring processing request, and obtain resolution information of video data to be processed in the video structuring processing request.

The calculating unit 42 is configured to calculate a video memory value consumed by processing the to-be-processed video data according to the resolution information of the to-be-processed video data, a preset resolution reference value, and a preset allocated video memory value.

The structural processing unit 43 is configured to perform video structural processing on the video data to be processed if the video memory value that needs to be consumed for processing the video data to be processed is less than or equal to the remaining video memory resources.

The pause unit 44 is configured to pause to respond to the video structured processing request if the video memory value consumed for processing the video data to be processed is greater than the remaining video memory resource, and to re-determine whether the video memory value consumed for processing the video data to be processed is greater than the remaining video memory resource after a preset time interval.

As an embodiment of the present application, the video structuring engine 40 further includes an initialization unit. Wherein:

the initialization unit is used for carrying out initialization operation.

As an embodiment of the present application, the initialization unit is specifically configured to: setting the maximum occupied video memory resource according to the size of the video card resource of the hardware acceleration equipment; setting the residual video memory resources according to the video memory resources occupied to the maximum extent; acquiring resolution information of each video source device; setting a preset resolution reference value according to the resolution information of each video source device; and setting a preset distribution video memory value according to the preset resolution reference value.

As an embodiment of the present application, the video structuring engine 40 further includes an updating unit. Wherein:

and the updating unit is used for updating the residual video memory resources in real time according to the video structuring processing progress.

As an embodiment of the present application, the updating unit is specifically configured to update the remaining video memory resources to the original remaining video memory resources minus a video memory value consumed when the video data to be processed is processed when the video structuring engine performs the structuring processing on the video data to be processed.

As an embodiment of the present application, the updating unit is further configured to update the remaining video memory resources to the original remaining video memory resources plus the video memory value consumed when processing the to-be-processed video data after the video structuring engine completes the data structuring processing.

As an embodiment of the present application, the structuring processing unit 43 includes: a decoding unit and a processing unit.

The decoding unit is used for decoding the video file in the video structuring processing request to obtain the video data to be processed.

The processing unit is used for carrying out structuralization processing on the video data to be processed.

It should be noted that, for the information interaction, execution process, and other contents between the above-mentioned devices/units, the specific functions and technical effects thereof are based on the same concept as those of the embodiment of the method of the present application, and specific reference may be made to the part of the embodiment of the method, which is not described herein again.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

It can be seen from the above that, the video structuring engine provided in this embodiment can also determine, based on the resolution information of the video data to be processed, the video memory resource that needs to be consumed when processing the video data by obtaining the resolution information of the video data to be processed, and process the video data to be processed only when the remaining video memory resources are sufficient, thereby implementing adaptive control on the video structuring concurrence path number of the multi-resolution video source, effectively avoiding the situation of video memory overflow, and maintaining the stability of the whole video structuring system when performing multi-resolution concurrence structuring.

Fig. 5 is a block diagram of a terminal device according to another embodiment of the present application. As shown in fig. 5, the terminal device 5 of this embodiment includes: a processor 50, a memory 51 and a computer program 52, such as a program of a video structuring method, stored in said memory 51 and being executable on said processor 50. The processor 50, when executing the computer program 52, implements the steps in the embodiments of the video structuring processing method described above, such as S11 to S14 shown in fig. 1. Alternatively, when the processor 50 executes the computer program 52, the functions of the units in the embodiment corresponding to fig. 4, for example, the functions of the units 41 to 44 shown in fig. 4, are implemented, for which reference is specifically made to the relevant description in the embodiment corresponding to fig. 4, which is not repeated herein.

Illustratively, the computer program 52 may be divided into one or more units, which are stored in the memory 51 and executed by the processor 50 to accomplish the present application. The one or more units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 52 in the terminal device 5. For example, the computer program 52 may be divided into a determination unit and an execution unit, and may also be divided into an acquisition unit, a scaling unit, a mapping unit and a training unit, and the specific functions of the units are as described above.

The terminal device may include, but is not limited to, a processor 50, a memory 51. Those skilled in the art will appreciate that fig. 5 is merely an example of a terminal device 5, and does not constitute a limitation of the terminal device 5, and may include more or fewer components than those shown, or some of the components may be combined, or different components, e.g., the terminal device may also include an input-output device, a network access device, a bus, etc.

The Processor 50 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 51 may be an internal storage unit of the terminal device 5, such as a hard disk or a memory of the terminal device 5. The memory 51 may also be an external storage device of the terminal device 5, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 5. Further, the memory 51 may also include both an internal storage unit and an external storage device of the terminal device 5. The memory 51 is used for storing the computer program and other programs and data required by the terminal device. The memory 51 may also be used to temporarily store data that has been output or is to be output.

The embodiments of the present application provide a computer program product, which when running on a mobile terminal, enables the mobile terminal to implement the steps in the above method embodiments when executed.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium may include at least: any entity or device capable of carrying computer program code to a photographing apparatus/terminal apparatus, a recording medium, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. Such as a usb-disk, a removable hard disk, a magnetic or optical disk, etc. In some jurisdictions, computer-readable media may not be an electrical carrier signal or a telecommunications signal in accordance with legislative and proprietary practices.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus/network device and method may be implemented in other ways. For example, the above-described apparatus/network device embodiments are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implementing, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present application, and not for limiting the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.

Claims

1. A video structuring method, comprising:

2. The video structuring processing method according to claim 1, further comprising:

an initialization operation is performed.

3. The video structuring processing method according to claim 2, wherein said performing an initialization operation includes:

setting the residual video memory resource according to the video memory resource occupied to the maximum extent;

acquiring resolution information of each video source device;

4. The video structuring processing method according to claim 1, further comprising:

5. The video structuring processing method according to claim 4, wherein the updating the remaining video memory resource in real time according to the video structuring processing progress includes:

when the video structuring engine carries out structuring processing on the video data to be processed, the residual video memory resource is updated to the original residual video memory resource minus the video memory value consumed when the video data to be processed is processed.

6. The method according to claim 4, wherein the updating the remaining video memory resource in real time according to the video structuring processing progress further comprises:

7. The video structural processing method according to any one of claims 1 to 6, wherein if the video memory value consumed for processing the video data to be processed is less than or equal to the remaining video memory resources, performing video structural processing on the video data to be processed includes:

and carrying out structuring processing on the video data to be processed.

8. A video structuring engine, comprising:

the calculation unit is used for calculating the video memory value consumed by processing the video data to be processed according to the resolution information of the video data to be processed, a preset resolution reference value and a preset distribution video memory value;

and the pause unit is used for pausing to respond to the video structured processing request if the video memory value consumed for processing the video data to be processed is greater than the residual video memory resource, and judging whether the video memory value consumed for processing the video data to be processed is greater than the residual video memory resource again after a preset time interval.

9. A terminal device, characterized in that the terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the terminal device, the processor implementing the steps of the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of a method according to any one of claims 1 to 7.