WO2022218001A1

WO2022218001A1 - Video analysis method and related system

Info

Publication number: WO2022218001A1
Application number: PCT/CN2022/072428
Authority: WO
Inventors: 谢奕; 陆瑞智; 李萍; 林雅珺; 袁晶
Original assignee: 华为云计算技术有限公司
Priority date: 2021-04-17
Filing date: 2022-01-17
Publication date: 2022-10-20

Abstract

Provided in the present application is a video analysis method, which is applied to the field of artificial intelligence (AI). The method comprises: acquiring a video analysis strategy, wherein the video analysis strategy is obtained according to a mining result of historical events that occur in a geographical region, and the video analysis strategy comprises a target time period and a target region, which require video analysis in the geographical region; then, acquiring target video data according to the video analysis strategy; and analyzing the target video data to obtain an analysis result. By means of the method, on the basis of an occurrence rule of historical events in a geographical region, a target region with a large event occurrence probability within a target time period is predicted, and key analysis is carried out on video data of the target region within the target time period. Therefore, reasonable distribution of an AI capability is realized, perceptual blind zones in terms of time and space dimensions are reduced, and efficient regional supervision is realized.

Description

Video analysis method and related system

technical field

The present application relates to the technical field of artificial intelligence (AI), and in particular, to a video analysis method, system, and computing device cluster, computer-readable storage medium, and computer program product.

Background technique

With the rapid development of artificial intelligence (AI) technology, regional governance based on AI capabilities has become a popular AI industry application.

For more refined regional governance, a large number of data collection devices, such as cameras, can usually be deployed in geographic areas. By collecting video, events that occur in a geographic area can be recorded. The video collected by a large number of cameras is the main source of perception of abnormal events in the city. The video analysis system can run one or more event analysis algorithms by invoking the underlying resources. For example, it can be an AI-based analysis algorithm, which analyzes and processes the video through the event analysis algorithm, so as to timely and automatically discover the events in the geographical area. Abnormal events, such as: traffic violations, urban management violations, emergency safety events, ecological and environmental protection events, etc.

However, due to the huge number of cameras deployed in a geographical area, using a video analysis system to analyze the videos generated by all cameras in a geographical area requires a lot of resources (for example, computing resources, storage resources, etc.), and the cost is too high , poor feasibility.

In order to reduce the resource consumption of video analysis in the video analysis system, some methods adopt the analysis method of periodically polling the videos captured by cameras in a geographical area, that is, divide all cameras in a geographical area into several batches, and analyze the video The system accesses different batches of cameras in turn according to time for video analysis. However, this method is prone to omission of key events, such as omission of violations on traffic roads.

How to enable the video analysis system to detect key events to a greater extent and achieve more efficient regional supervision under the circumstance of limited resources has become an urgent problem to be solved in the current regional governance based on AI capabilities.

SUMMARY OF THE INVENTION

The present application provides a video analysis method, which can realize efficient regional supervision based on the occurrence law of historical events in each region. The present application also provides a video analysis system, a computing device cluster, a computer-readable storage medium, and a computer program product corresponding to the above method.

In a first aspect, the present application provides a video analysis method. The method is performed by a video analysis system. In some embodiments, the video analysis system may be a software system. The computing device or cluster of computing devices executes the video analysis method by running the program code of the software system. In other embodiments, the video analysis system may also be a hardware system for video analysis.

Specifically, the video analysis system obtains a video analysis strategy, and the video analysis strategy is obtained according to the mining results of historical events that occurred in the geographical area. The video analysis strategy includes the target period and target area in which video analysis needs to be performed in the geographical area, and then the video analysis system The target video data is acquired according to the video analysis strategy, and the target video data records the condition of the target area in the geographical area during the target period, and then the video analysis system analyzes the target video data to obtain the analysis result.

In this method, the video analysis system predicts the target area with a high probability of event occurrence in the target period based on the occurrence law of historical events in the geographical area, and then analyzes the video data of the target area in the target period, and realizes the reasonable AI capability. distribute. By allocating AI capabilities to a reasonable time and place, it can adapt to complex and changeable environments, reduce perception blind spots in time and space dimensions, improve the detection rate of key events, and improve the effect of regional governance. Moreover, the video analysis system does not need to analyze the full amount of video data, and applies limited resources to target time periods and target areas with high event probability, which improves resource utilization and reduces the cost of video analysis.

In some possible implementation manners, the video analysis system may obtain information of historical events occurring in a geographic area, perform mining and analysis on the information of historical events, and obtain mining results. Wherein, the mining result includes the event occurrence probability of each area in the geographic area during the target period. The video analysis system then generates video analysis strategies for geographic regions based on the mining results.

In this method, the occurrence law of events reflected by the information of historical events occurring in the geographical area can provide auxiliary information for video analysis, for example, assist the video analysis system to analyze the video data of the target area during the target time period with high probability of event occurrence. Through analysis, the detection rate of key events can be improved, thereby improving the effect of regional governance.

In some possible implementations, the video analytics policy also includes event types. Based on this, the video analysis system can also determine the event analysis algorithm corresponding to the event type according to the event type in the video analysis strategy, and then use the event analysis algorithm to analyze the target video data to obtain the analysis result.

By analyzing the target video data through the event analysis algorithm corresponding to the event type, it is possible to avoid the occurrence of missing key events caused by video analysis using unmatched algorithms or general algorithms, and to improve the detection rate of key events.

In some possible implementations, the video analysis system may generate a video analysis job based on target video data and an event analysis algorithm, and then schedule the video analysis job to the target computing node to obtain analysis results after the target computing node executes the video analysis job.

The method schedules the video analysis job to the target computing node for execution, and the target computing node can perform the video analysis to ensure the normal operation of the video analysis service.

In some possible implementations, the target computing node may be a terminal device, a device in an edge environment, or a device in a cloud environment. The video analysis system can schedule video analysis jobs to corresponding devices according to business requirements. For example, a video analysis system can schedule video analysis jobs to devices close to the video source to reduce transmission overhead and latency.

In some possible implementations, the video analysis system may also obtain information on computing resources. The information of computing resources includes the maximum concurrent number of computing resources. The computing resources include a collection of computing nodes distributed in various places. The maximum number of video analysis jobs that this set allows to execute in parallel is the above-mentioned maximum concurrent number. Based on this, the video analysis system may update the video analysis strategy according to the maximum concurrent number. In this way, the computing resources can be fully utilized and the waste of computing resources can be avoided.

In some possible implementations, when the number of target areas that need to perform video analysis during the target period is greater than the maximum concurrent number, the video analysis system may Event occurrence probability, update the target area that needs to perform video analysis in the target time period. The updated number of the target regions is less than or equal to the maximum concurrent number. For example, the updated target area may be a plurality of areas with the occurrence probability of the above-mentioned event at the top in descending order.

In this way, limited computing resources can be preferentially used to perform video analysis on a target area with a high event probability, which improves the detection rate of key events and improves the utilization rate of computing resources.

In some possible implementations, the event occurrence probability of each area in the geographic area during the target period may include the event occurrence probability of each event type of each area in the geographic area during the target period. For example, the event occurrence probability of the A region of the geographic region during the time period T1 may include the C1 type event occurrence probability and the C2 type event occurrence probability of the A region during the time period T1. Based on this, the video analysis system can also obtain the event type corresponding to the above target area according to the event occurrence probability of each event type in each area in the geographical area within the target time period. The updated video analytics policy also includes the above event types.

In this way, an event analysis algorithm corresponding to the event type can be determined based on the event type corresponding to the target area, and event analysis can be performed in a targeted manner.

In some possible implementations, when the analysis result indicates that an event occurs in the target area within the target period, the video analysis system may further report the analysis result to the event processing system. Intelligent and automated regional governance can be realized to meet business needs.

In some possible implementations, the video analysis system may preprocess historical work order data in a geographic area to obtain information on historical events that occurred in the geographic area. The historical event information includes multiple pieces of work order information, and each piece of work order information includes time, location, and event type.

In this method, the video analysis system can process unstructured historical work order data into structured work order information through AI technologies such as natural language processing, image recognition or speech recognition, which provides a basis for work order mining. In turn, it provides help for video analysis based on work order mining.

In some possible implementations, the historical work order data of a geographic area is derived from one or more of the following data: events reported by personnel, events of manual inspection, and historical events obtained by the video analysis system analysis. Among them, diversified historical work order data can reduce system errors, help to improve the accuracy of work order mining, and then improve the accuracy of video analysis based on work order mining.

In some possible implementations, the historical events analyzed by the video analysis system include verified historical events. By verifying the historical events analyzed by the video analysis system, the validity of work order mining based on the historical events can be guaranteed.

In a second aspect, the application provides a video analysis system, which includes the following functional modules:

The strategy management module is used to obtain a video analysis strategy, the video analysis strategy is obtained according to the mining results of historical events that occurred in the geographical area, and the video analysis strategy includes the target period and target area in which the video analysis needs to be performed in the geographical area ;

A video analysis module, configured to obtain target video data according to the video analysis strategy, the target video data records the status of the target area in the geographical area during the target period, and analyzes the target video data to obtain the analysis results.

In some possible implementations, the video analysis system further includes:

A work order mining module, configured to acquire information of historical events occurring in the geographical area, perform mining analysis on the information of the historical events, and obtain mining results, wherein the mining results include each area in the geographical area the probability of occurrence of the event during said target period;

The work order mining module is further configured to generate the video analysis strategy for the geographical area according to the mining result.

In some possible implementations, the video analysis strategy further includes an event type; the video analysis module is specifically used for:

Determine an event analysis algorithm corresponding to the event type according to the event type in the video analysis strategy;

Use the event analysis algorithm to analyze the target video data to obtain an analysis result.

In some possible implementations, the video analysis module is specifically used for:

Generate a video analysis job based on the target video data and the event analysis algorithm;

scheduling the video analysis job to the target computing node;

Obtain an analysis result after the target computing node performs the video analysis job.

In some possible implementations, the target computing node is a terminal device, a device in an edge environment, or a device in a cloud environment.

In some possible implementations, the policy management module is also used for:

Obtain information about computing resources, where the information about computing resources includes the maximum concurrent number of computing resources;

The video analysis policy is updated according to the maximum concurrent number.

In some possible implementations, the policy management module is specifically used for:

When the number of target areas for which video analysis needs to be performed in the target period is greater than the maximum concurrent number, update the video that needs to be performed in the target period according to the event occurrence probability of each area in the geographical area in the target period The analyzed target area, the updated number of the target area is less than or equal to the maximum concurrent number.

In some possible implementations, the video analysis module is also used for:

When the analysis result indicates that an event occurs in the target area within the target period, the analysis result is reported to the event processing system.

In some possible implementations, the work order mining module is specifically used for:

Preprocessing the historical work order data of the geographical area to obtain information of historical events occurring in the geographical area, wherein the information of the historical events includes multiple pieces of work order information, and each piece of work order information includes time, location and event type.

In some possible implementations, the historical work order data of the geographic area is derived from one or more of the following data:

The events reported by personnel, the events of manual inspection, and the historical events obtained by the analysis of the video analysis system.

In a third aspect, the present application provides a video analysis method, which can be performed by a video analysis system. Specifically, the video analysis system obtains historical work order data of a geographic area, and according to the historical work order data, obtains mining results of historical events that occurred in the geographic area, and the mining results include that each area of the geographic area is in The event occurrence probability of the target period, and then according to the event occurrence probability of each area of the geographical area in the target period, the target video data is obtained, and the target video data records the target area in the geographical area in the target period. condition, the target area is an area where the event occurrence probability satisfies a preset condition, and then the target video data is analyzed to obtain an analysis result.

It should be noted that, for the beneficial effects produced by executing the method, reference may be made to the description of the relevant content of the first aspect, which will not be repeated here.

In some possible implementations, the event occurrence probability of each area of the geographic area during the target time period includes the probability that various types of events occur in each area of the geographic area during the target time period. The video analysis system may also obtain the event type corresponding to the target area according to the probability of occurrence of various types of events in each area of the geographic area during the target period. Correspondingly, the video analysis system may determine an event analysis algorithm corresponding to the event type according to the event type corresponding to the target area, and use the event analysis algorithm to analyze the target video data to obtain an analysis result.

In some possible implementations, the video analysis system may generate a video analysis job based on the target video data and the event analysis algorithm, schedule the video analysis job to a target computing node, obtain the target computing node to execute the Analysis results after a video analysis job.

In some possible implementations, the video analysis system may also acquire information on computing resources, where the information on computing resources includes the maximum concurrent number of computing resources. Correspondingly, the video analysis system may acquire the target video data according to the event occurrence probability of each area of the geographical area in the target period and the maximum concurrent number. The target video data records the condition of the target area in the geographic area during the target period, and the number of the target area is less than or equal to the maximum concurrent number. Wherein, the target area may be a plurality of areas in the target period in which the event occurrence probability is ranked from the largest to the smallest.

In a fourth aspect, the present application provides a video analysis system. The video analysis system includes various modules or units for executing the video analysis method in the third aspect or any possible implementation manner of the third aspect.

In a fifth aspect, the present application provides a computing device cluster, where the computing device cluster includes at least one computing device. At least one computing device includes at least one processor and at least one memory. The processor and the memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory to cause a cluster of computing devices to perform the method of the first aspect or any implementation of the first aspect.

In a sixth aspect, the present application provides a computer-readable storage medium, where instructions are stored in the computer-readable storage medium, and the instructions instruct a computing device or a cluster of computing devices to execute the first aspect or any one of the first aspects. implement the method described.

In a seventh aspect, the present application provides a computer program product comprising instructions that, when run on a computing device or a cluster of computing devices, cause the computing device or cluster of computing devices to perform the first aspect or any one of the first aspects above implement the method described. On the basis of the implementation manners provided by the above aspects, the present application may further combine to provide more implementation manners.

Description of drawings

In order to illustrate the technical methods of the embodiments of the present application more clearly, the following briefly introduces the accompanying drawings used in the embodiments.

1 is a schematic diagram of a video analysis method based on timing polling provided by an embodiment of the present application;

2A is a schematic diagram of an application scenario of a video analysis method provided by an embodiment of the present application;

FIG. 2B is a schematic diagram of the deployment of a video analysis system according to an embodiment of the present application;

3 is a flowchart of a video analysis method provided by an embodiment of the present application;

4 is a schematic diagram of a work order database provided by an embodiment of the present application;

5A is a schematic flowchart of mining and analysis based on a knowledge graph according to an embodiment of the present application;

FIG. 5B is another schematic flowchart of mining and analysis based on a knowledge graph according to an embodiment of the present application;

6 is a schematic diagram of a work order map constructed based on work order information according to an embodiment of the present application;

7 is a flowchart of another video analysis method provided by an embodiment of the present application;

FIG. 8 is a flowchart of a verification experiment provided by the embodiment of the present application;

9A is a schematic diagram of an experimental result provided by an embodiment of the present application;

9B is a schematic diagram of another experimental result provided by the embodiment of the application;

FIG. 10 is a schematic structural diagram of a computing device cluster according to an embodiment of the present application.

Detailed ways

The terms "first" and "second" in the embodiments of the present application are only used for the purpose of description, and cannot be understood as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature.

In order to facilitate the understanding of the embodiments of the present application, first, some terms involved in the present application are explained.

Artificial intelligence (AI) is a technical discipline that simulates human thought processes and/or intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) through computer programs running on computers. Specifically, artificial intelligence can be realized through an AI model. An AI model is an algorithm that can learn rules from massive data. The AI model after targeted training through massive data can have specific functions.

Artificial intelligence includes multiple branches, for example, including different branches such as computer vision (computer vision, CV), natural language processing (natural language processing, NLP), speech recognition (automatic speech recognition, ASR).

Computer vision is a technology that uses cameras and computers to replace human eyes, so that computers have the functions of segmenting, classifying, identifying, detecting or making event discriminative decisions. Computer vision technology can be implemented through algorithms (that is, through AI models). For example, a camera deployed in a factory can capture the video of the production environment, and the video frame of the video can be input to the flame detection model. The features of the video frames are extracted by the flame detection model, and learning based on the features can determine whether the production environment exists. The flame detection model is used to determine whether an emergency safety event such as a fire occurs. The flame detection model is an AI model trained with a pre-prepared data set. Another example: a camera deployed on the road can collect road video, and the video frame of the road video can be input to the target detection model, which can detect vehicles on the road, etc., and then determine whether traffic violations such as illegal parking have occurred. Among them, Object detection models are AI models trained on pre-prepared datasets.

In practical applications, computer vision can be used for intelligent governance of geographic areas. Specifically, a large number of cameras are deployed in a geographic area, and the video analysis system invokes the underlying computing resources to run one or more event analysis algorithms based on computer vision to analyze and process the video collected by the cameras to discover traffic violations in the geographic area. Incidents, urban management violations, emergency security incidents and other abnormal events, so as to achieve intelligent regional governance.

Currently, data collection devices such as cameras deployed in geographic areas can collect massive amounts of video, and a large amount of computing resources and storage resources are consumed for full analysis of the aforementioned massive amounts of video. To this end, the related art adopts the analysis method of timed polling. Specifically, a geographic area may be divided into multiple different areas, and in different time periods of a cycle (eg, one day), the video analysis system takes turns accessing cameras in different areas to perform video analysis. For example, referring to FIG. 1, the video analysis system can access the cameras in area 1 during the T1 period, analyze the video source C, access the cameras in the area 2 during the T2 period, analyze the video source B, and access the area during the T3 period. The camera of 3 analyzes the video source A.

However, the environment of geographical areas is usually complex and changeable. The video analysis system analyzes the preset area in each time period, which is difficult to match the complex and changeable environment, and it is easy to form perception blind spots in the time and space dimensions, resulting in critical Event omission. For example, in the above example, during the T3 period, there is a high probability of events in area 1, and the video analysis system is connected to the camera in area 3 to analyze video source A at this time, but the camera in area 1 is not connected to video source C. analysis, resulting in key event omissions. On the one hand, the omission of key events will affect the effect of regional governance. On the other hand, the application of resources (such as computing resources and storage resources for video analysis) in time periods and regions with a low probability of occurrence of events results in waste of resources.

In view of this, embodiments of the present application provide a video analysis method. The method may be performed by a video analysis system. In some embodiments, the video analysis system may be a software system. The computing device or cluster of computing devices executes the video analysis method by running the program code of the software system. In other embodiments, the video analysis system may also be a hardware system for video analysis. The embodiments of the present application use a video analysis system as a software system for illustration.

Specifically, the historical work order data of a geographical area can reflect historical events that occurred in the geographical area, and the video analysis system can mine and analyze the historical events that occurred in the geographical area based on the historical work order data, such as mining the temporal and spatial distribution of historical events. Analysis, and then based on the mining results, a video analysis strategy is obtained. The video analysis strategy includes target time periods and target areas in the geographic area for which video analysis is required. The video analysis system obtains target video data based on the above video analysis strategy, the target video data records the video data of the target area in the geographical area in the target period, and then the video analysis system analyzes the above target video data to obtain analysis results.

In this method, the video analysis system predicts the target area with a high probability of event occurrence in the target period based on the occurrence law of historical events in the geographical area, such as the temporal and spatial distribution of historical events, and analyzes the video data of the target area in the target period. , to achieve a reasonable allocation of AI capabilities (for example, the ability to deploy computing resources for AI algorithms such as event analysis algorithms). By allocating AI capabilities to a reasonable time and place, it can adapt to complex and changeable environments, reduce perception blind spots in time and space dimensions, improve the detection rate of key events, and improve the effect of regional governance. Moreover, the video analysis system does not need to analyze the full amount of video data, and applies limited resources to target time periods and target areas with high event probability, which improves resource utilization and reduces the cost of video analysis.

It should be noted that this method is not a video analysis method developed for a specific business, and can be widely applied to various video analysis scenarios. For example, it can be applied to traffic management scenarios to detect traffic violations such as speeding and running red lights, or to safety management scenarios to detect emergency safety events such as fire and smoke, or to ecological environment protection scenarios to prevent damage Environmental incidents such as stealing sewage, incineration of garbage, etc. are detected.

In order to make the technical solutions of the present application clearer and easier to understand, the following describes the application scenarios of the present application with reference to the accompanying drawings.

Referring to the schematic diagram of the application scenario of the video analysis method shown in FIG. 2A , the scenario includes a multi-channel video source 100 , a video management platform 200 , a video analysis system 300 , and a service management platform 400 . The multi-channel video source 100 is connected to the video management platform 200 , and the video analysis system 300 is connected to the video management platform 200 and the service management platform 400 respectively.

The video source 100 may include any one or more of terminal-side video streams (camera video streams), edge video streams, cloud video streams, and offline video files. Similar to the video analysis system 300, in some embodiments, the video management platform 200 and the service management platform 400 may be software systems, and computing devices or computing device clusters run the program codes of the above software systems to perform video management and service management ( such as regional governance). In other embodiments, the video management platform 200 may also be a hardware system for video management, and the service management platform 400 may also be a hardware system for service management. FIG. 2A uses the video management platform 200 , the video analysis system 300 , and the service management platform 400 as software systems for illustration.

The user can interact with the video management platform 200 , the video analysis system 300 , and the service management platform 400 respectively through the client to realize functions such as video management, video analysis, and service management. The client may be a general client that integrates video management functions, video analysis functions, and service management functions, or may be multiple independent clients with video management functions, video analysis functions, and service management functions, respectively. The client may be a web client or an application client, which is not limited in this embodiment of the present application.

Specifically, the video management platform 200 includes an interaction module 201 and a video management module 202 . The interaction module 201 is used for accessing the multi-channel video sources 100, and the video management module 202 is used for unified management of the multi-channel video sources. For example, the video management module 202 supports the user to view pictures of multiple video sources 100 in batches. For another example, the video management module 202 supports the video analysis system 300 to obtain target video data through a unified application programming interface (application programming interface, API) interface.

The video analysis system 300 is a unified platform for video analysis and processing. For different abnormal event detection requirements in different application scenarios, the video analysis system 300 may provide corresponding video analysis methods. Specifically, the video analysis system 300 may include a work order mining module 301 , a policy management module 302 and a video analysis module 303 .

The work order mining module 301 is configured to perform mining analysis on the information of historical events occurring in the geographical area to obtain mining results. The information of historical events occurring in the geographic area may be multiple pieces of work order information obtained by preprocessing historical work order data. Each ticket information includes time, location and event type. The mining results include the event occurrence probability of each area in the geographic area during the target period. Wherein, the target period may be one or more periods in a cycle.

The strategy management module 302 is configured to obtain a video analysis strategy according to the mining result. The video analysis strategy includes target time periods and target areas in the geographic area for which video analysis is required. Mining results can also include event types. Correspondingly, the video analysis strategy can also indicate the event types of events that may occur in the target area during the target period.

The video analysis module 303 is configured to acquire target video data according to the video analysis strategy, analyze the target video data, and obtain an analysis result. In some embodiments, the video analytics policy also includes an event type. The video analysis module 303 may determine an event analysis algorithm corresponding to the above event type according to the event type in the video analysis strategy, and use the event analysis algorithm to analyze the target video data to obtain an analysis result.

Wherein, when the video analysis module 303 analyzes the target video data, it can generate a video analysis job based on the target video data and the event analysis algorithm, and then schedule the video analysis job to the target computing node, for example, combining computing The resource distribution schedules the video analysis job to the corresponding target computing node, and obtains the analysis result after the target computing node executes the video analysis job.

In some possible implementations, the video analysis system 300 also includes an interaction module 304 . The interaction module 304 is used to support the user to perform management operations on the video analysis job through a user interface such as a graphical user interface (graphical user interface, GUI), a command user interface (command user interface, CUI), or an API interface, such as video analysis. A job performs a view, start, or stop operation.

The service management platform 400 is a unified platform for processing events, and may also be called an event processing system. When the analysis result of the video analysis system 300 indicates that an event occurs in the target area within the target period, the video analysis system 300 may report the analysis result to the service management platform 400, so that the service management platform 400 can The event is processed. Specifically, the analysis result may be reported to the service management platform 400 in the form of a work order. The work order includes at least one of the following work order information: time, location, and event type.

The service management platform 400 may include an interaction module 401 and an event processing module 402 . The interaction module 401 is used to support the user to verify the work order reported by the video analysis system 300 through a user interface such as GUI or CUI. Further, the interaction module 401 can be used to support the user to correct the work order when an error occurs in the work order reported by the video analysis system 300 . The event processing module 402 is configured to process the event corresponding to the work order reported by the video analysis system 300 according to the work order information. Specifically, the event processing module 402 may perform processes such as allocating, disposing, and filing the work order. For example, when the event processing module 402 receives a work order for a fire, it can send a reminder message to the fire department according to the time and location in the work order, so as to remind the fire department to dispatch fire trucks nearby to perform fire fighting tasks.

It should be noted that, in the embodiment shown in FIG. 2A , the video analysis system 300 performs work order mining, generates a video analysis strategy based on the work order mining, and illustrates an example of performing video analysis according to the video analysis strategy. The video analysis system 300 shown in FIG. 2A implements the work order mining function through the work order mining module 301 . The above-mentioned work order mining module 301 may be a software module or a hardware module of the video analysis system 300 . In some possible implementations, the video analysis system 300 may not include the above-mentioned work order mining module 301 , and the work order mining function may be implemented by a work order mining system independent of the video analysis system 300 .

Similarly, the video analysis system 300 may not include the above-mentioned policy management module 302, and the policy management function may be implemented by a policy management system independent of the video analysis system 300, or by a work order mining system with a policy management function.

The interaction process between the video analysis system 300 and other platforms has been described in detail above. Next, the deployment method of the video analysis system 300 is illustrated by way of example.

In some possible implementations, as shown in FIG. 2B , when the video analysis system 300 is a software system, the video analysis system 300 may be deployed on one or more computing devices (eg, a central server) in a cloud environment. In some embodiments, when the video analysis system 300 is a hardware system, the video analysis system 300 may include one or more computing devices on a cloud environment. The cloud environment indicates a cluster of central computing devices owned by the cloud service provider and used to provide computing, storage, and communication resources. Accordingly, the video analysis system 300 may provide a cloud service of video analysis for the user to use.

In specific implementation, the user can trigger the operation of starting the video analysis system 300 through the client. The video analysis system 300 can obtain the target video data from the video management platform 200 according to the video analysis strategy, and then the video analysis system 300 can analyze the target video data. Analysis, for example, generates a video analysis job based on target video data and an event analysis algorithm, and then schedules the event analysis job to the target computing node. The target computing node may be a terminal device as shown in FIG. 2B , or a computing node in a cloud environment, or a computing node in an edge environment.

The terminal devices include but are not limited to user terminals such as desktop computers, notebook computers, and smart phones, or camera terminals with computing capabilities, such as software-defined cameras (Software-Defined Camera, SDC). A cloud environment indicates a cluster of central computing devices owned by a cloud service provider for providing computing, storage, and communication resources. It should be noted that the cloud environment where the target computing node is located and the cloud environment where the video analysis system 300 is located may be the same cloud environment or different cloud environments. FIG. 2B uses the above cloud environment as the same cloud environment for illustration. The edge environment indicates a cluster of edge computing devices that are geographically close to the terminal device (ie, the terminal-side device) and are used to provide computing, storage, and communication resources.

The user can also trigger the operation of viewing, starting or stopping the video analysis job through the client. The video analysis system 300 can display the execution progress and execution result of the video analysis job to the user in response to the user's viewing operation, or respond to the user's start. operation, start the corresponding video analysis job, and stop executing the corresponding video analysis job in response to the user's stop operation.

In other possible implementations, the video analysis system 300 may also be deployed in an edge environment, for example, on one or more computing devices (edge computing devices) in the edge environment, or the video analysis system 300 includes an edge environment one or more computing devices in the . Edge computing devices can be servers, computing boxes, and the like. Of course, the video analysis system 300 may also be deployed on a terminal device, or be a terminal device used for video analysis. This embodiment does not limit this.

The video analysis system 300 can not only be centrally deployed in a cloud environment, an edge environment or a terminal device. Considering that the video analysis system 300 can be divided into multiple parts, for example, into multiple functional modules, the above-mentioned multiple functional modules can also be deployed in different environments in a distributed manner.

The deployment mode of the video analysis system 300 is described above by way of example, and the deployment mode of the video management platform 200 and the service management platform 400 can be referred to the deployment mode of the video analysis system 300, and details are not repeated here.

Next, from the perspective of the video analysis system 300, the video analysis method provided by the embodiments of the present application is introduced with reference to the accompanying drawings.

Referring to the flowchart of the video analysis method shown in FIG. 3, the method includes the following steps:

S302: The video analysis system 300 acquires information of historical events that occurred in the geographic area.

Specifically, the video analysis system 300 may acquire historical work order data of the geographical area, and acquire information of historical events occurring in the geographical area according to the historical work order data. The information of the historical event includes multiple pieces of work order information, and each piece of work order information includes time, location, and time type.

Wherein, the historical work order data of the geographic area comes from one or more of the following data: events reported by personnel, events of manual inspection, and historical events analyzed by the video analysis system 300 . The above historical work order data is usually unstructured data, such as work order images reported manually, work order images analyzed by the video analysis system 300, or work order text and work order voices reported by personnel. Based on this, the video analysis system 300 may preprocess the historical work order data of the geographic area, so as to obtain information of historical events that occurred in the geographic area. The information of the historical event is structured data.

According to different types of work order history data, the video analysis system 300 may adopt different preprocessing methods. For example, when the historical work order data is a work order image, the video analysis system 300 can identify the work order image through an image recognition algorithm, thereby obtaining information of historical events that occurred in a geographic area. For another example, when the historical work order data is work order text, the video analysis system 300 can extract key information through technologies such as NLP, so as to obtain information of historical events that occurred in the geographic area. Also for example, when the historical work order data is the work order voice, the video analysis system 300 can recognize the work order voice through ASR and other technologies, obtain the work order text, and then extract key information from the work order text through NLP and other methods, so as to obtain the work order text. Information on historical events that occurred in a geographic area.

Further, the video analysis system 300 may store the information of historical events occurring in the geographic area, for example, in the work order database, so as to facilitate further mining and analysis based on the information of the historical events in the future. For ease of understanding, the embodiments of the present application also provide corresponding examples for description.

Referring to the schematic diagram of the work order database shown in FIG. 4 , as shown in FIG. 4 , the work order database includes multiple pieces of work order information, and each piece of work order information includes time, location, and event type. Wherein, time may be the time of occurrence. The reporting time and the occurrence time are roughly the same, or the difference is small, which can be ignored or not, or it can be the reporting time. The time in the ticket information can generally be accurate to minutes or seconds. The location can be accurate to the district (county level) of the city plan, and in some embodiments, the location can also be accurate to the building or the house number of the building. Event types may include event categories and event subcategories, wherein the event category may be, for example, environmental sanitation, greening and lighting, city appearance order, and the like, and the event subcategory may be, for example, psoriasis, unclean roads, and the like.

As shown in FIG. 4 , the work order information may further include the work order source. In some embodiments, the source of the work order may include manual inspection (for example, grid personnel report after inspection), personnel report (for example, citizens report via hotline, mailbox, etc.), AI analysis (for example, video analysis system) 300 Analysis) any one or more. Further, the work order information may also include other auxiliary information, such as event description information.

S304: The video analysis system 300 performs mining analysis on the information of the historical event to obtain a mining result.

Specifically, the video analysis system 300 may select at least one data analysis method, such as at least one of numerical statistics, knowledge graph, or reinforcement learning, to mine and analyze the information of historical events to obtain mining results. The mining result includes the event occurrence probability of each area in the geographic area during the target period.

For ease of understanding, examples of mining analysis based on numerical statistics and mining analysis based on knowledge graphs are described below.

In a first implementation manner, the video analysis system 300 can count the occurrence frequency of events in each region in the geographical area in each time period in the past period of time, and estimate the event occurrence probability of each region in the geographical area in the target time period based on the event occurrence frequency .

Further, the video analysis system 300 can also count the occurrence frequency of a certain type of event in each region in the geographical area in the past period of time according to the event type, and then count different events based on the occurrence frequency of different types of events. Type of event probability.

The following description will be given in conjunction with a specific example. In this example, the past period of time can be T days, and each day can be divided into 24 time periods by hour. The work order database provides historical work order information for T days. The video analysis system 300 can determine each area in the geographic area by the following formula In each period, the frequency of each type of event:

Wherein, p(n, t, i) is the frequency of events of event type i in the t-th period of the day under video source n, 1≤n≤N, 1≤t≤24. I(T', n, t, i) is a marking function, indicating whether an event of event type i occurs in the t-th period of the day under the video source n in the work order of day T', if Has occurred, its value is 1, otherwise it is 0.

When the amount of data is large enough, the video analysis system 300 can approximate the occurrence frequency of the event as the event occurrence probability, so as to obtain the probability distribution Φ of events of different event types in space and time. In this example, the probability distribution Φ is an N×24×C matrix, where C is the number of event types, and each element in the matrix is used to characterize the event occurrence probability of an event type in a region in a time period . The video analysis system 300 can obtain the event occurrence probability of each region in the geographic region in the target time period according to the above probability distribution Φ.

In the second implementation manner, the video analysis system 300 may construct a work order map, perform mining analysis based on the work order map, and then obtain the event occurrence probability of each region in the geographical area in the target period. The video analysis system 300 can predict the occurrence trend of a specific event when mining and analyzing the work order map, so as to obtain the event occurrence probability of the specific event. The video analysis system 300 can also integrate the eventual relationship of different events (for example, the eventual logical relationship between events, such as succession, causation, condition, and upper and lower levels) based on the work order map, and predict the occurrence trend of each hot event, thereby. Obtain the event occurrence probability of each hot event.

Referring to a schematic flowchart of mining and analysis based on a knowledge graph shown in FIG. 5A , after extracting work order information from historical work order data, the video analysis system 300 can construct a knowledge graph based on the work order information. A knowledge graph is a graph structure data composed of several vertices connected to each other according to rules. Among them, the knowledge graph constructed based on the work order information is also called the work order graph. The ticket graph takes events as vertices, and each vertex includes at least three attributes including time, location and event type. The vertices are connected to each other according to the time, place, and event type association.

The video analysis system 300 can extract different features such as time series features, spatial sequence features, and related event features from the work order map. Predict the trend of that particular event in an area. As shown in FIG. 5A , the occurrence trend of the specific event is represented by a curve, and the curve is used to represent the event occurrence probability of the specific event in a certain area under different time periods. Compared with the analysis method based on numerical statistics, this method introduces different map features and makes predictions based on more and richer features, which can improve the prediction accuracy and provide more help for AI capability scheduling.

Referring to another schematic flowchart of mining and analysis based on a knowledge graph shown in FIG. 5B , after extracting work order information from historical work order data, the video analysis system 300 can construct a work order graph based on the work order information. The video analysis system 300 can then mine dynamic hot events in different regions based on the work order map, obtain a spatiotemporal dynamic graph model of event correlation, and extract time series features, spatial sequence features, and related temporal features from the work order map. Then, the event-related spatiotemporal dynamic graph model is integrated with time series features, spatial sequence features and related time features, and the trained prediction model is input to predict the event occurrence probability of different event types.

The prediction model can be a graph neural network (GNN). Among them, the GNN model can include various types, and FIG. 5B is illustrated with a temporal Graph Convolutional Networks model. The prediction model can output the event occurrence probability of each event type. In this method, the GNN-based prediction model can integrate the eventual relationship of different events and multi-dimensional spatiotemporal attributes for prediction, which further improves the accuracy.

For ease of understanding, the present application also provides a specific example for description.

Referring to the schematic diagram of the work order map shown in FIG. 6 , in this example, a water accumulation event occurred at location A in a certain city at 8:00, and a vertex of the work order map can be constructed based on the water accumulation event. Attributes can include the following key-value pairs: (location, A), (time, 8 o'clock), (event type, road surface water). Then, at location A, at 9 o'clock and 10 o'clock, the water accumulation event still exists, so the other two vertices of the work order map can be constructed. Since these three flood events are related in time, place and event type, the above three vertices can be connected.

Location A is adjacent to location B, and location B also had a water accumulation event at 9:00. Due to the correlation of location, time, and event type, the apex of the water accumulation event at location B is also the same as the one at location A. Vertex association. In addition, the ground water caused a traffic jam event at location A at 9 o'clock. Due to the correlation between location and time, the vertices of the traffic jam event were also associated with the vertices of the three water accumulation events at location A.

Based on the connection relationship of the vertices in the above graph, the continuity of events in time series, the correlation in geographical location, and the impact on other events can be obtained. Based on this, the video analysis system can extract time-series features, spatial-sequence features, and related event features from the atlas, and input the prediction model for prediction to obtain the probability of event occurrence. For example, when predicting the probability of a water accumulation event at 11:00, the occurrence of water accumulation at this location A at 8:00, 9:00, and 10:00 can be obtained as the time series feature of the event. The location A and location B are adjacent, and the occurrence of the water accumulation event at location B can be obtained as the spatial sequence feature of the event. The water accumulation event is related to other events (such as traffic jam events), so the occurrence of other events at 8:00, 9:00, and 10:00 at location A can be obtained as the relevant event features. By integrating the above features into a feature vector and inputting the trained prediction model, the probability of a water accumulation event at 11 o'clock in location A can be obtained.

S306: The video analysis system 300 generates the video analysis strategy for the geographical area according to the mining result.

The video analysis strategy includes a target time period and a target area in the geographic area for which video analysis needs to be performed. For example, the target time period and target area may be the time period and area corresponding to the elements in the probability distribution Φ whose event occurrence probability is greater than the preset threshold. In some possible implementations, the video analysis strategy also includes event types. The event type may be, for example, an event type corresponding to an element in the probability distribution Φ whose event occurrence probability is greater than a preset threshold.

The video analysis system 300 may determine the target area according to the event occurrence probability of each area of the geographic area in the mining result in the target period, and generate a video analysis strategy based on the target period and the target area. Further, when the event occurrence probability in the mining result includes the event occurrence probability of each event type, the video analysis system 300 may generate a target event type corresponding to the target area according to the target period, target area, and event type (for example, the target event type corresponding to the target area). Video analytics strategies for geographic regions.

In some possible implementations, the video analysis system 300 may also obtain information on computing resources. Among them, the computing resource is a set including computing nodes distributed in various places. Compute nodes in this set can compute concurrently. In this application, the maximum number of video analysis jobs that the set allows to execute in parallel is referred to as the maximum concurrent number of computing resources. The information of computing resources includes the maximum concurrent number of computing resources. The video analysis system 300 may also update the video analysis policy according to the maximum concurrent number.

Specifically, when the number of target areas for which video analysis needs to be performed in the target period is greater than the maximum concurrent number, the video analysis system 300 may update the target period according to the event occurrence probability of each area in the geographical area during the target period. Target area for video analysis. The number of updated target regions is less than or equal to the maximum concurrent number above. Wherein, the updated target area may be the area with the above-mentioned probability in descending order. In some embodiments, the updated target area includes the top M areas in descending order of probability. Among them, M is the maximum concurrent number.

Similarly, the event occurrence probability of each area in the geographical area during the target period includes the event occurrence probability of each event type, that is, when the probability of each type of event occurring in each area in the target period, the video analysis system 300 can also be based on the geographical area. The probability of occurrence of various types of events in each area in the target period is obtained, and the event type corresponding to the updated target area is obtained. The updated video analytics policy also includes this event type.

For ease of understanding, the following description is given with examples. The video analysis system 300 can, according to the probability distribution Φ, select the top M area-event type combinations with the highest probability in each time period of the day, and generate a video analysis strategy, which indicates that in the corresponding time period, according to the highest probability before the There are M area-event type combinations, and the target video data of the corresponding area is analyzed to determine whether an event of the corresponding type occurs. The video analysis strategy can also be expressed as a matrix Ψ, which can also be called an AI capability scheduling matrix.

Among them, the matrix Ψ is an N×24×C matrix, and the value of each element in the matrix is 0 or 1, which is used to indicate whether to schedule the AI capability of the corresponding event type in the corresponding time period and corresponding area, that is, whether to The corresponding video data is analyzed to determine whether an event of the corresponding event type occurs.

In some possible implementations, part of the historical work order data may come from historical events analyzed by the video analysis system 300 . Considering that the video analysis system 300 has false detections, the service management platform 400 can also analyze the video analysis system 300 The reported analysis results are verified to ensure the validity of mining and analysis of historical event information in historical work order data. In other possible implementation manners, the video analysis system 300 may also set an abnormal termination mechanism to ensure the validity of mining and analysis of historical event information in historical work order data, thereby ensuring the accuracy of the video analysis strategy.

The abnormal termination mechanism may be: when the termination condition is satisfied, the historical events analyzed by the video analysis system 300 are excluded. Mining and analysis can be performed based on information on remaining historical events, such as events reported by personnel and events manually inspected. In some embodiments, the specific historical events analyzed by the video analysis system 300 may also be excluded, and mining analysis is performed based on the information of the remaining historical events.

The termination condition may be that the scheduling frequency of at least one area-event type combination is greater than a preset frequency. The scheduling frequency may be the ratio of the scheduling duration (duration of scheduling the corresponding AI capability) of the above region-event type combination within a day (eg, the past day) to the duration of one day. The preset frequency can be set according to the experience value, for example, it can be set to 70%. Further, the termination condition may further include a preset ratio where the number of region-event type combinations whose scheduling frequency is greater than the preset frequency is less than the preset ratio of the maximum concurrent number of computing resources. The preset ratio can be set according to the experience value, for example, it can be set as 30%. ,

For ease of understanding, this application also provides an example. For example, the scheduling policy for the past day can be characterized by the matrix Ψ'. Each element in the matrix Ψ' is used to indicate whether the AI capability corresponding to the corresponding event type is scheduled for the corresponding time period of the corresponding area. For example, the value of the element is 1, indicating that the corresponding AI capability is scheduled, and the value of the element is 0, which means that the corresponding AI capability is not scheduled. The video analysis system 300 can determine the scheduling frequency of the area-event type combination through the matrix Ψ', and count the area-event type combination whose scheduling frequency is greater than a preset frequency, such as 70%. When the number is less than or equal to a preset ratio of the maximum concurrent number of computing resources, such as 30%, the scheduling policy of the day can remain unchanged. When the number is greater than the preset ratio of the maximum concurrent number of computing resources, the abnormal termination mechanism can be triggered, and the scheduling policy of the day can be updated. This can solve the problem that AI algorithms such as event analysis algorithms continue to generate misjudged events and miss real events, improving usability.

It should be noted that the above S302 to S306 are an optional implementation manner for the video analysis system 300 to obtain the video analysis strategy in this embodiment. In other possible implementation manners of this embodiment, the work order mining system may The information is mined and analyzed, the analysis result is obtained, and the video analysis strategy is obtained based on the analysis result. Video analytics system 300 may obtain video analytics policies from the work order mining system.

S308: The video analysis system 300 acquires target video data according to the video analysis strategy.

Specifically, the video analysis strategy includes a target area and a target time period. The video analysis system 300 can determine the target video source from the multi-channel video sources 100 accessed by the video management platform 200 according to the target area, and then obtain the target time period from the target video source. video data to obtain the target video data.

Further, when the video analysis system 300 obtains the AI capability scheduling matrix Ψ representing the video analysis strategy based on the distribution of computing resources, the target video data can also be obtained from the video management platform 200 according to the AI capability scheduling matrix. For example, the video analysis system 300 can schedule the time period and area corresponding to the element whose element value is 1 in the matrix Ψ according to the AI capability, and obtain the video data of the area in the target time period, thereby obtaining the target video data.

In some possible implementations, after obtaining the mining results, the video analysis system 300 may directly obtain the target video data according to the event occurrence probability of each region in the geographic region in the mining results in the target period. The target video data records the condition of the target area in the geographic area during the target period. The target area may be an area where the event occurrence probability satisfies a preset condition. In some embodiments, the preset condition may be that the event occurrence probability is greater than a preset threshold. In other embodiments, the preset condition may be the top M in order of event occurrence probability in descending order. Of course, the preset condition may also be that the event occurrence probability is greater than the preset threshold, and the top M is sorted from largest to smallest.

S310: The video analysis system 300 determines an event analysis algorithm corresponding to the event type according to the event type in the video analysis strategy.

Specifically, when an event type is included in the video analysis strategy, the video analysis system 300 may determine an event analysis algorithm corresponding to the event type from a variety of event analysis algorithms, so as to analyze the target video data.

Similarly, when the video analysis system 300 obtains the AI capability scheduling matrix Ψ representing the video analysis strategy in combination with the distribution of computing resources, it can also determine the type of event corresponding to the element whose value is 1 in the AI capability scheduling matrix. The corresponding event analysis algorithm.

It should be noted that, the foregoing S308 and S310 may be executed in parallel, or may be executed sequentially according to the set order, which is not limited in this embodiment of the present application. In addition, the foregoing S310 is an optional step of this embodiment of the present application, and the above-mentioned S310 may not be executed to execute the video analysis method of this embodiment of the present application. For example, when the video analysis strategy does not include event types, or the user is concerned about whether a certain event type occurs, but does not care about whether other event types occur, the above-mentioned S310 may not be performed.

S312: The video analysis system 300 generates a video analysis job based on the target video data and the event analysis algorithm.

Specifically, the video analysis system 300 can directly package the target video data and the code package of the event analysis algorithm to generate a video analysis job. In some embodiments, the video analysis system 300 may also package the target video data and the identification of the event analysis algorithm to generate a video analysis job. The identifier of the event analysis algorithm may include the name of the event analysis algorithm, a hash code, and the like. The identifier of the event analysis algorithm is used to instruct the execution node of the video analysis job to obtain the corresponding event analysis algorithm and analyze the target video data.

S314: The video analysis system 300 schedules the video analysis job to the target computing node.

Specifically, the video analysis system 300 may schedule the above-mentioned video analysis jobs respectively to execution nodes of the jobs, and the execution nodes are called target computing nodes. The target computing node may be a terminal device, that is, a terminal computing node, a computing node in a cloud environment, or a computing node in an edge environment.

Considering the delay problem, the video analysis system 300 can first determine the idle computing nodes, and then select the nearest M computing nodes from the idle computing nodes as the target computing nodes, and then schedule the video analysis job to the above-mentioned target computing nodes, In order to facilitate the target computing node to perform the above video analysis job.

S316: The video analysis system 300 obtains the analysis result after the target computing node performs the video analysis job. When the analysis result indicates that an event occurs in the target area within the target period, S318 is executed.

Specifically, when the target computing node executes the video analysis job, it can execute the code package of the event analysis algorithm in the video analysis job to run the event analysis algorithm. Get analysis results.

In some possible implementations, the target computing node can also obtain the code package of the event analysis algorithm according to the identifier of the event analysis algorithm in the video analysis job. The target computing node obtains the code package of the event analysis algorithm through the network-connected storage device), and then executes the code package of the event analysis algorithm to run the event analysis algorithm, and the target video data in the video analysis job is processed through the event analysis algorithm. analysis, the analysis results can be obtained.

The video analysis system 300 may receive the analysis result returned by the target computing node. When the analysis result indicates that an event occurs in the target area within the target period, that is, when an event is detected in the target video data, the video analysis system 300 may perform the analysis result reporting process, as shown in S318.

The foregoing S312 to S316 are an implementation manner in which the video analysis system 300 in this embodiment of the present application analyzes the target video data and obtains an analysis result. In other possible implementation manners of the embodiments of the present application, the video analysis system 300 may also analyze the target video data in other ways. For example, the video analysis system 300 may directly call an event analysis algorithm to analyze the target video data. This embodiment does not limit this.

S318: The video analysis system 300 reports the analysis result to the service management platform 400, so that the service management platform 400 processes the event.

Specifically, the video analysis system 300 may report the analysis result to the service management platform 400 in the form of a work order (also referred to as an intelligent inspection work order). Among them, the video analysis system 300 provides a unified output interface. The video analysis system 300 reports the work order to the service management platform 400 through the unified output interface, so that the service management platform 400 can process the event corresponding to the work order. For example, the business management platform 400 can automatically perform processes such as allocating, filing, and disposing of work orders. Further, the service management platform 400 also supports the user to manage the work order through a GUI interface or the like, or to correct the work order that is wrongly analyzed by the video analysis system 300 . The business management platform 400 can also preprocess the work orders output by the video analysis system 300 to obtain work order information, and then update the work order information to the work order database, so as to improve the effect of mining and analysis by using the continuously updated work order database.

The foregoing S318 is an optional step in this embodiment of the present application, and the above step may not be performed when the video analysis method is executed, which is not limited in this embodiment of the present application.

Based on the above description, the video analysis system 300 predicts the target area with a high probability of event occurrence within the target time period based on the occurrence law of historical events in the geographical area, such as the temporal and spatial distribution of historical events, and analyzes the video data of the target area in the target time period. Focused on analysis to achieve a reasonable distribution of AI capabilities. By allocating AI capabilities to a reasonable time and place, it can adapt to complex and changeable environments, reduce detection blind spots, improve the detection rate of key events, and improve the effect of regional governance. Furthermore, the video analysis system 300 can apply limited resources to target time periods and target areas with a high probability of event occurrence, thereby improving resource utilization and reducing analysis costs.

The embodiment shown in FIG. 3 is mainly exemplified by the video analysis system 300 performing work order mining, video analysis strategy, and video analysis based on the video analysis strategy. The present application also provides another embodiment of a video analysis method. In this embodiment, work order mining may be implemented by the work order mining system 500 .

Referring to the flowchart of the video analysis method shown in FIG. 7 , the method is completed by the video management platform 200 , the video analysis system 300 , the business management platform 400 and the work order mining system 500 in cooperation. Specifically, the method includes the following steps:

Step 1: The preprocessing system (not shown in FIG. 7 ) acquires historical work order data, preprocesses the historical work order data, obtains multiple pieces of work order information, and stores the multiple pieces of work order information in the work order database.

The historical work order data can be derived from any one or more of the following: historical events reported by grid personnel, historical events reported by citizens through hotlines, mailboxes, e-mails, etc., and historical events analyzed by the video analysis system 300 (also known as smart patrol events). The intelligent inspection events may include events manually corrected by the user.

For the process of preprocessing the historical work order data by the preprocessing system, reference may be made to the description of the relevant content of the embodiment shown in FIG. 3 , for example, reference to the description of the relevant content of S302 , which will not be repeated here.

Step 2: The work order mining system 500 obtains multiple pieces of work order information from the work order database, performs mining analysis on the multiple pieces of work order information, and obtains mining results.

Wherein, the mining result includes the event occurrence probability of each area in the geographic area during the target period. The event occurrence probability of each region in the geographic area during the target period can represent the spatiotemporal distribution law of the event.

For the process of mining and analyzing the work order information by the work order mining system 500, reference may be made to the relevant content description of the mining and analysis performed by the video analysis system 300 in the embodiment shown in FIG.

Step 3: The work order mining system 500 obtains a round-robin scheduling scheme based on work order mining according to the mining result.

In this embodiment, the polling scheduling scheme based on work order mining is a preliminary scheduling scheme, and the scheduling scheme is equivalent to the preliminary video analysis strategy generated by the video analysis system 300 based on the mining results in the embodiment shown in FIG. 3 . Based on this, the work order mining system 500 obtains, according to the mining result, the specific implementation of the round-robin scheduling scheme based on work order mining, please refer to the description of the relevant content of S306.

Step 4: The video analysis system 300 acquires computing resource distribution information, and performs AI algorithm intelligent scheduling according to the computing resource distribution information to obtain an optimized scheduling scheme.

The optimized scheduling scheme refers to an updated video analysis strategy obtained by updating the video analysis strategy in combination with the computing resource distribution information. For the specific implementation thereof, reference may be made to the description of the related content of S306, which will not be repeated here.

Step 5: The video analysis system 300 obtains target video data according to the optimized scheduling scheme.

Specifically, the video source 100 accesses the video management platform 200 for unified management, and the video analysis system 300 can obtain target video data from the video management platform 200 according to the optimized scheduling scheme, for example, video data of the target area in the target time period.

Step 6: The video analysis system 300 starts intelligent video analysis on the target video data based on the optimized scheduling scheme.

Specifically, the optimized scheduling scheme also includes an event type, and the video analysis system 300 can determine a corresponding event analysis algorithm based on the event type, and then can generate a video analysis job according to the target video data and the event analysis algorithm, and use the video analysis job to Scheduled to target computing nodes, such as end-side camera terminals or edge servers, cloud servers (ie, cloud servers), etc., the target computing nodes can run event analysis algorithms to analyze target video data and obtain analysis results. The video analysis system 300 can obtain the analysis result of the target computing node.

Step 7: When the analysis result indicates that an event occurs in the target area during the target period, the video analysis system 300 may output the intelligent inspection event.

The video analysis system 300 may report the intelligent inspection event to the service management platform 400 in the form of an intelligent inspection event work order.

Step 8: The business management platform 400 receives the intelligent inspection event work order, and performs work order allocation, processing and filing.

Among them, the service management platform 400 also supports the user to perform manual verification on the intelligent inspection event work order. When there is an error in the intelligent inspection event work order, the service management platform 400 may also receive the user's correction to the intelligent inspection event work order. Correspondingly, the business management platform 400 can perform work order allocation, processing and filing according to the revised work order.

In this embodiment, the work order mining system 500 can mine the work order information in the work order database, and generate a preliminary video analysis strategy based on the mining results. The video analysis system 300 can further optimize according to the preliminary video analysis strategy, and then Intelligent video analysis is realized based on the optimized video analysis strategy. When the video analysis system 300 detects an event, it can be reported in the form of a work order. The work order information of the work order can be updated into the work order database to further optimize the work order mining system 500 .

The embodiment of the present application also designs a control experiment to verify the effect of the video analysis method of the embodiment of the present application and the video analysis method in the related art.

The experimental design scheme is: simulating the scale of a real geographic area, assuming that 10,000 cameras are deployed in the geographic area, and a day includes 24 time periods. There are 3 types of events that can occur in each time period at each location. Since computing resources are usually limited, only concurrent identification of events in N video sources can usually be supported at the same time, where N<10000·3.

As shown in Figure 8, it is assumed that the real event occurrence probability follows a Gaussian-like distribution P in space and time. Based on this distribution P, a series of historical ticket data can be randomly generated. The period of the historical work order data can be set according to actual needs. In this example, historical ticket data for the past 10 days can be generated. Among them, each work order data contains three kinds of information: time, location, and event type. Considering that the work order data can come from either manual reporting or intelligent identification by the video analysis system 300, and there is a certain error in the identification result of the video analysis system 300. Therefore, 50% of the work order data can be selected as the work order data recognized by the video analysis system 300. In the work order data recognized by the video analysis system 300, the event type of each work order data has a 20% false positive rate. There is an 80% recognition accuracy rate. Based on the above historical work order data, the probability distribution of each type of event in space and time can be predicted by numerical statistics. Then, in each period, the top M location-event type combinations with the highest probability can be determined according to the maximum concurrent number M of computing resources, and the AI capability scheduling matrix can be generated according to the top M location-event type combinations with the highest probability.

In the testing phase, a series of work order data can be randomly generated as a test set following the distribution P. Then, the event capture rate and resource waste rate are determined according to the test set and the AI capability scheduling matrix. The specific calculation formulas are as follows:

Among them, the number of events in the test set can be regarded as the total number of real events. When an event of the corresponding event type actually occurs in the area and time period corresponding to the element whose value is 1 in the AI capability scheduling matrix, the AI resource can be correctly covered to Add 1 to the number of events, and after traversing all elements in the AI capability scheduling matrix whose value is 1, the number of events correctly covered by AI resources can be obtained. The AI resources may be computing resources for running AI algorithms such as event analysis algorithms.

Similarly, the total number of AI resources deployed can be M. When the area and time period corresponding to the element value of 1 in the AI capability scheduling matrix does not actually occur an event of the corresponding event type, the number of AI resources not covered by the event can be calculated. Add 1, and after traversing all the elements whose value is 1 in the AI capability scheduling matrix, the number of AI resources that are not covered by the event can be obtained.

FIGS. 9A and 9B show the experimental results of the video analysis method (also referred to as the statistics-based scheduling method), the analysis method based on timed polling, and the video analysis method based on full access of the present application, as shown in FIGS. 9A and 9B . As shown in 9B, compared with the video analysis method based on timed polling, the method of the present application can achieve a higher event capture rate and a lower resource waste rate under the same concurrent resource cost, and when the concurrent resources are more When the time limit is reached, the advantage of resource waste rate is more obvious. In addition, although the video analysis method based on full access can achieve a 100% event capture rate, the resource waste rate is as high as 91%, the cost is high, and the availability is low. Whether it is a video analysis method based on timed polling or a video analysis method based on full access, they are essentially mechanical scheduling based on a fixed duration, which results in the application of a large number of AI resources to unnecessary locations, making it difficult to adapt Changes in the environment have resulted in a lot of waste of resources. The method of the embodiment of the present application, on the other hand, performs mining and analysis based on work order data, and based on the analysis results, can realize reasonable scheduling of AI capabilities or AI resources, and improve resource utilization.

Based on the video analysis method provided by the embodiments of the present application, the embodiments of the present application further provide a video analysis system 300 as described above. The video analysis system provided by the embodiments of the present application will be introduced below with reference to the accompanying drawings.

Referring to the schematic structural diagram of the video analysis system 300 shown in FIG. 2A , the system 300 includes: a policy management module 302 and a video analysis module 303 . Wherein, the strategy management module 302 is specifically configured to execute S306 in the aforementioned method flow of Fig. 3, and the video analysis module 303 is configured to execute S308 in the aforementioned method flow of Fig. 3 to obtain target video data, and then analyze the target video data, Get analysis results.

In some possible implementations, the policy management module 302 may not perform the above S306. For example, the video analysis system 300 may further include a work order mining module 301, and the work order mining module 301 may execute the above S306, and correspondingly, the policy management module 302 may directly obtain the video analysis policy. For another example, the video analysis system 300 is connected to the work order mining system 500 , the work order mining system 500 can generate a video analysis strategy according to the mining result, and the video analysis system 300 can obtain the video analysis strategy from the work order mining system 500 .

When the video analysis system 300 further includes a work order mining module 301, the work order mining module 301 can also be used to perform S302 to S304 in the aforementioned method flow of FIG. The mining results of single information generate video analysis strategies for geographic areas.

In some possible implementations, the video analysis module 303 is specifically configured to perform S310 in the aforementioned method flow of FIG. 3 to determine an event analysis algorithm corresponding to the event type, and then analyze the target video data according to the event analysis algorithm to obtain Analyze the results.

In some possible implementations, the video analysis module 303 is specifically configured to execute S312 to S316 in the aforementioned method flow of FIG. 3 to generate a video analysis job, and schedule the video analysis job to the target computing node, so as to realize the analysis of target video data. Analyze and get the results of the analysis.

In some possible implementation manners, the target computing node may be a terminal device, a device in an edge environment, or a device in a cloud environment, which is not limited in this embodiment.

In some possible implementations, the policy management module 302 is further configured to execute S306 in the foregoing method flow in FIG. 3 , so as to update the video analysis policy according to the maximum concurrent number of computing resources.

In some possible implementations, the video analysis module 303 is further configured to execute S318 in the foregoing method flow in FIG. 3 , so as to report the analysis result to an event processing system, such as the service management platform 400 .

In some possible implementation manners, the work order mining module 301 is specifically configured to execute S302 in the foregoing method flow of FIG. 3 to obtain information of historical events occurring in a geographic area.

In some possible implementations, the historical ticket data for the geographic area is derived from one or more of the following data:

The video analysis system 300 according to the embodiments of the present application may correspond to executing the methods described in the embodiments of the present application, and the above-mentioned and other operations and/or functions of the respective modules/units of the video analysis system 300 are implemented in order to realize the implementation shown in FIG. 3 , respectively. For the sake of brevity, the corresponding flow of each method in the example will not be repeated here.

Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device, and any computing device in the at least one computing device may be from a cloud environment or an edge environment, or may be a terminal device. The computing device cluster is specifically used to implement the functions of the video analysis system 300 in the embodiment shown in FIG. 2A .

FIG. 10 provides a schematic structural diagram of a computing device cluster. As shown in FIG. 10 , the computing device cluster 10 includes multiple computing devices 1000 , and the computing devices 1000 include a bus 1001 , a processor 1002 , a communication interface 1003 and a memory 1004 . Communication between the processor 1002 , the memory 1004 and the communication interface 1003 is through the bus 1001 .

The bus 1001 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 10, but it does not mean that there is only one bus or one type of bus.

The processor 1002 may be a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc. any one or more of the devices.

The communication interface 1003 is used for external communication. For example, the communication interface 1003 is used to output analysis results and the like.

Memory 1004 may include volatile memory, such as random access memory (RAM). The memory 1004 may also include non-volatile memory (non-volatile memory), such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (solid state drive) , SSD).

Computer-readable instructions are stored in the memory 1004, and the processor 1002 executes the computer-readable instructions to cause the computing device cluster 10 to perform the aforementioned video analysis method (or implement the aforementioned functions of the video analysis system 300).

Specifically, in the case of implementing the embodiment of the system shown in FIG. 2A, and the functions of each module of the video analysis system 300 described in FIG. 2A, such as the work order mining module 301, the policy management module 302, and the video analysis module 303, are: In the case of software implementation, the software or program code required to perform the functions of the modules in FIG. 2A may be stored in at least one memory 1004 in the computing device cluster 10 . At least one processor 1002 executes program code stored in memory 1004 to cause computing device cluster 10 to perform the aforementioned video analysis method.

Embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium may be any available medium that a computing device can store, or a data storage device such as a data center that contains one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVDs), or semiconductor media (eg, solid state drives), and the like. The computer-readable storage medium includes instructions instructing a computing device or cluster of computing devices to perform the video analysis method described above.

The embodiments of the present application also provide a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on the computing device, all or part of the processes or functions described in the embodiments of the present application are generated. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be transmitted from a website site, computing device or data center via Transmission to another website site, computing device, or data center by wired (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.) means. The computer program product can be a software installation package, which can be downloaded and executed on a computing device or a cluster of computing devices if any one of the aforementioned video analysis methods needs to be used.

The descriptions of the processes or structures corresponding to each of the above-mentioned drawings have their own emphasis, and for the parts that are not described in detail in a certain process or structure, reference may be made to the related descriptions of other processes or structures.

Claims

A video analysis method, wherein the method is applied to a video analysis system, comprising:

Obtaining a video analysis strategy, the video analysis strategy is obtained according to the mining results of historical events occurring in the geographical area, and the video analysis strategy includes a target period and a target area in which video analysis needs to be performed in the geographical area;

Acquiring target video data according to the video analysis strategy, the target video data records the status of the target area in the geographic area during the target period;

The target video data is analyzed to obtain an analysis result.
The method according to claim 1, wherein before acquiring the video analysis strategy, the method further comprises:

obtain information on historical events that occurred in the geographic area;

Mining and analyzing the information of the historical event to obtain a mining result, wherein the mining result includes the event occurrence probability of each area in the geographical area in the target time period;

The video analysis strategy for the geographic area is generated according to the mining result.
The method according to claim 1 or 2, wherein the video analysis strategy further includes an event type; the analyzing the target video data to obtain an analysis result, comprising:

Determine an event analysis algorithm corresponding to the event type according to the event type in the video analysis strategy;

Use the event analysis algorithm to analyze the target video data to obtain an analysis result.
method according to claim 3, is characterized in that, described utilizes described event analysis algorithm to analyze described target video data, obtains analysis result, specifically comprises:

Generate a video analysis job based on the target video data and the event analysis algorithm;

scheduling the video analysis job to the target computing node;

Obtain an analysis result after the target computing node performs the video analysis job.
The method according to claim 4, wherein the target computing node is a terminal device, a device in an edge environment or a device in a cloud environment.
The method according to any one of claims 1 to 5, wherein the method further comprises:

Obtain information about computing resources, where the information about computing resources includes the maximum concurrent number of computing resources;

The video analysis policy is updated according to the maximum concurrent number.
The method according to claim 6, wherein the updating the video analysis policy according to the maximum concurrent number comprises:

When the number of target areas for which video analysis needs to be performed in the target period is greater than the maximum concurrent number, update the video that needs to be performed in the target period according to the event occurrence probability of each area in the geographical area in the target period The analyzed target area, the updated number of the target area is less than or equal to the maximum concurrent number.
The method according to any one of claims 1 to 7, wherein the method further comprises:

When the analysis result indicates that an event occurs in the target area within the target period, the analysis result is reported to the event processing system.
The method according to claim 2, wherein the acquiring information of historical events occurring in the geographical area comprises:

Preprocessing the historical work order data of the geographical area to obtain information of historical events occurring in the geographical area, wherein the information of the historical events includes multiple pieces of work order information, and each piece of work order information includes time, location and event type.
The method according to claim 9, wherein the historical work order data of the geographical area is derived from one or more of the following data:

The events reported by personnel, the events of manual inspection, and the historical events obtained by the analysis of the video analysis system.
A video analysis system, comprising:

The strategy management module is used to obtain a video analysis strategy, the video analysis strategy is obtained according to the mining results of historical events that occurred in the geographical area, and the video analysis strategy includes the target period and target area in which the video analysis needs to be performed in the geographical area ;

A video analysis module, configured to obtain target video data according to the video analysis strategy, the target video data records the status of the target area in the geographical area during the target period, and analyzes the target video data to obtain the analysis results.
The system of claim 11, wherein the video analysis system further comprises:

A work order mining module, configured to acquire information of historical events occurring in the geographical area, perform mining analysis on the information of the historical events, and obtain mining results, wherein the mining results include each area in the geographical area the probability of occurrence of the event during said target period;

The work order mining module is further configured to generate the video analysis strategy for the geographical area according to the mining result.
The system according to claim 11 or 12, wherein the video analysis strategy further includes an event type; the video analysis module is specifically used for:

Determine an event analysis algorithm corresponding to the event type according to the event type in the video analysis strategy;

Use the event analysis algorithm to analyze the target video data to obtain an analysis result.
system according to claim 13, is characterized in that, described video analysis module is specially used for:

Generate a video analysis job based on the target video data and the event analysis algorithm;

scheduling the video analysis job to the target computing node;

Obtain an analysis result after the target computing node performs the video analysis job.
The system according to claim 14, wherein the target computing node is a terminal device, a device in an edge environment or a device in a cloud environment.
The system according to any one of claims 11 to 15, wherein the policy management module is further configured to:

Obtain information about computing resources, where the information about computing resources includes the maximum concurrent number of computing resources;

The video analysis policy is updated according to the maximum concurrent number.
The system according to claim 16, wherein the policy management module is specifically configured to:

When the number of target areas for which video analysis needs to be performed in the target period is greater than the maximum concurrent number, update the video that needs to be performed in the target period according to the event occurrence probability of each area in the geographical area in the target period The analyzed target area, the updated number of the target area is less than or equal to the maximum concurrent number.
The system according to any one of claims 11 to 17, wherein the video analysis module is further configured to:

When the analysis result indicates that an event occurs in the target area within the target period, the analysis result is reported to the event processing system.
The system according to claim 12, wherein the work order mining module is specifically used for:

Preprocessing the historical work order data of the geographical area to obtain information of historical events occurring in the geographical area, wherein the information of the historical events includes multiple pieces of work order information, and each piece of work order information includes time, location and event type.
The system according to claim 19, wherein the historical work order data of the geographic area is derived from one or more of the following data:

The events reported by personnel, the events of manual inspection, and the historical events obtained by the analysis of the video analysis system.
A computing device cluster, characterized in that the computing device cluster includes at least one computing device, the at least one computing device includes at least one processor and at least one memory, and the at least one memory stores computer-readable instructions, the at least one processor executing the computer-readable instructions, causing the cluster of computing devices to perform the method of any one of claims 1-10.
A computer-readable storage medium, characterized by comprising computer-readable instructions, when the computer-readable instructions are executed on a computing device or a cluster of computing devices, the computing device or the cluster of computing devices is made to perform as claimed in claim 1 The method of any one of to 10.
A computer program product, characterized in that it includes computer-readable instructions that, when the computer-readable instructions are executed on a computing device or a cluster of computing devices, cause the computing device or the cluster of computing devices to perform as claimed in claims 1 to 10 The method of any one.