[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN112183587A - Offshore wind farm biological monitoring method and system, computer equipment and storage medium - Google Patents

Offshore wind farm biological monitoring method and system, computer equipment and storage medium Download PDF

Info

Publication number
CN112183587A
CN112183587A CN202010946992.2A CN202010946992A CN112183587A CN 112183587 A CN112183587 A CN 112183587A CN 202010946992 A CN202010946992 A CN 202010946992A CN 112183587 A CN112183587 A CN 112183587A
Authority
CN
China
Prior art keywords
biological
node
model
recognition model
identification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010946992.2A
Other languages
Chinese (zh)
Other versions
CN112183587B (en
Inventor
董超
陈焱琨
周德富
王伟平
蒋俊杰
欧阳永忠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South China Sea Survey Technology Center State Oceanic Administration (south China Sea Marine Buoy Center)
Original Assignee
South China Sea Survey Technology Center State Oceanic Administration (south China Sea Marine Buoy Center)
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South China Sea Survey Technology Center State Oceanic Administration (south China Sea Marine Buoy Center) filed Critical South China Sea Survey Technology Center State Oceanic Administration (south China Sea Marine Buoy Center)
Priority to CN202010946992.2A priority Critical patent/CN112183587B/en
Publication of CN112183587A publication Critical patent/CN112183587A/en
Application granted granted Critical
Publication of CN112183587B publication Critical patent/CN112183587B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/25Fusion techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/94Hardware or software architectures specially adapted for image or video understanding
    • G06V10/95Hardware or software architectures specially adapted for image or video understanding structured as a network, e.g. client-server architectures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/52Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00Speaker identification or verification techniques
    • G10L17/18Artificial neural networks; Connectionist approaches
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L17/00Speaker identification or verification techniques
    • G10L17/26Recognition of special voice characteristics, e.g. for use in lie detectors; Recognition of animal voices
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/27Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique
    • G10L25/30Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 characterised by the analysis technique using neural networks
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Human Computer Interaction (AREA)
  • Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Acoustics & Sound (AREA)
  • Computational Linguistics (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Biology (AREA)
  • Software Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Signal Processing (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

The application relates to a method, a system, a computer device and a storage medium for offshore wind farm biological monitoring, wherein the method comprises the following steps: uploading biological information obtained by a biological identification end for biological identification in each node area to a service terminal; adjusting the node area according to the biological information distribution after data enhancement of each biological recognition end, creating a node server according to the adjusted node area, and issuing an initial biological recognition model to the corresponding biological recognition end through the node server; and each node server side carries out iterative training on each initial biological recognition model through a gradient descent method, uploads the trained initial biological recognition models to the server side, and carries out model fusion on each trained initial biological recognition model through a fusion algorithm by the server side so as to obtain the target biological recognition model. The invention can avoid improving the accuracy of biological identification and carry out reliability monitoring on species distribution, abundance and habitat of the organisms in the sea area of the wind power plant.

Description

Offshore wind farm biological monitoring method and system, computer equipment and storage medium
Technical Field
The application relates to the technical field of marine ecological environment monitoring, in particular to a method, a system, computer equipment and a storage medium for monitoring biology of an offshore wind farm.
Background
With the continuous development of ocean construction, offshore wind power application is generated. Offshore wind power plays an important role in practical application as one of important directions for development and utilization of renewable energy. However, offshore wind power development has great influence on marine ecology due to disturbance and noise generated during construction, construction and operation. Monitoring of the active area of marine organisms has therefore become of paramount importance.
At present, the monitoring methods for the distribution, abundance and habitat of the biological species in the sea area of the offshore wind farm comprise video monitoring and passive acoustic monitoring. However, since the moving area of marine life is difficult to capture, the image of the target living body cannot be recorded completely, and the biological sound production signal is recorded from a complex noise environment. Therefore, the coverage area capable of monitoring is small, timeliness is poor, accuracy is low, and long-term monitoring cannot be carried out.
Disclosure of Invention
Therefore, in order to solve the technical problems, a method, a system, a computer device and a storage medium for marine wind farm biological monitoring are provided, so that the accuracy of biological identification is improved, and the reliability monitoring on the distribution, abundance and habitat of marine organism species in the wind farm is realized.
In a first aspect, an embodiment of the present invention provides an offshore wind farm biological monitoring method, which is applied to a wind farm sea area, wherein a plurality of node areas are arranged in the wind farm sea area, each node area is provided with an information acquisition device group and a biological identification terminal, and the method includes the following steps:
uploading biological information obtained by a biological identification end for biological identification in each node area to a service terminal;
the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side;
each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model.
Further, before the step of issuing the initial biometric model to the corresponding biometric terminal through the node server, the method further includes: and the service terminal trains the initial biological recognition model according to the loss function and the biological information after data enhancement until the loss value of the model tends to converge.
Further, the method further comprises;
receiving biological information of a current node area monitored by the information acquisition equipment group, wherein the biological information comprises a video stream and an audio stream;
and respectively carrying out classification marking on the image frames and the audio frames in the video stream and the audio stream to obtain a biological and ship image data set and a biological and background sound data set.
Further, the method further comprises:
according to a convolutional neural network and a first activation function corresponding to the neural network, carrying out forward propagation on data in a biological and ship image data set which is input into the convolutional neural network in a divided mode to obtain a biological and ship prediction image, calculating a first difference value between the biological and ship prediction image and a standard biological and ship image according to a first loss function, carrying out backward propagation on the first difference value, and carrying out iterative training in a preset period to obtain a biological and ship image identification model;
according to a BP neural network and a second activation function corresponding to the neural network, carrying out forward propagation on data in biological and background sound data sets input into the BP neural network in batches to obtain a biological and background sound prediction frame, calculating a second difference value between the biological and background sound prediction frame and a standard biological and background sound frame according to a second loss function, carrying out backward propagation on the second difference value, and carrying out iterative training in a preset period to obtain a biological and background sound recognition model;
and integrating the biological and ship image recognition model and the biological and background sound recognition model to obtain an initial biological recognition model.
Further, the method for performing model fusion on each trained initial biometric model by the server through a fusion algorithm includes:
receiving an initial biological recognition model obtained after iterative training of each node server;
and taking each trained initial biological recognition model as input, and integrating each trained initial biological recognition model through a federal average algorithm to obtain a target biological recognition model.
Further, the method further comprises: and issuing the target biological identification model to each biological identification terminal for biological identification, performing static security analysis according to an identification result, and adding biological information of which the identification accuracy is smaller than a standard value into biological identification constraint so as to constrain the target biological model.
Further, the biological information acquisition equipment group comprises a video acquisition equipment group and an audio acquisition equipment group,
the video acquisition equipment set comprises a plurality of first video acquisition equipment arranged above water and a plurality of second video acquisition equipment correspondingly arranged below water, and the first video acquisition equipment is associated with the second video acquisition equipment;
the audio acquisition equipment group comprises a plurality of hydrophone arrays of equipment corresponding to the second video acquisition equipment, and the hydrophone arrays are associated with the second video acquisition equipment.
On the other hand, the embodiment of the invention also provides an offshore wind farm biological monitoring system, which is applied to the sea area of a wind farm, wherein a plurality of node areas are arranged in the sea area of the wind farm, each node area is provided with an information acquisition equipment group and a biological identification terminal, and the system comprises:
the information acquisition module is used for uploading biological information acquired by a biological identification terminal for biological identification in each node area to the service terminal;
the node creating module is used for enabling the server to issue data enhancement information according to the biological information distribution of each biological identification terminal, adjusting the node area according to the biological information distribution after the data enhancement, creating a node server according to the adjusted node area, and issuing an initial biological identification model to the corresponding biological identification terminal through the node server;
and the model generation module is used for enabling each node server to carry out iterative training on each initial biological recognition model through a gradient descent method, uploading the trained initial biological recognition models to the server, and carrying out model fusion on each trained initial biological recognition model through a fusion algorithm by the server so as to obtain a target biological recognition model.
An embodiment of the present invention further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the following steps when executing the computer program:
uploading biological information obtained by a biological identification end for biological identification in each node area to a service terminal;
the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side;
each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model.
An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the following steps:
uploading biological information obtained by a biological identification end for biological identification in each node area to a service terminal;
the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side;
each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model.
According to the method, the system, the computer equipment and the storage medium for the biological monitoring of the offshore wind farm, biological information obtained by a biological recognition end for biological recognition in each node area can be uploaded to a service terminal; the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side; each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model. According to the method, the initial biological recognition models corresponding to the node servers are recognized and trained through biological information in each node area in the wind power sea area, and the trained initial biological recognition models are subjected to model fusion through a fusion algorithm, so that the obtained biological recognition models have high robustness, the reliability monitoring on the distribution, abundance and habitat of the biological species in the sea area of the wind power plant is realized, and the actual application requirements are met.
Drawings
FIG. 1 is a schematic flow chart of a method for monitoring biology of an offshore wind farm according to an embodiment of the present invention;
FIG. 2 is a schematic view of the detailed process of step S102 in FIG. 1;
FIG. 3 is a block diagram of a biological monitoring system of an offshore wind farm according to an embodiment of the present invention;
fig. 4 is a block diagram of a node creation module according to an embodiment of the present invention;
fig. 5 is an internal structural diagram of a computer device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The application provides an offshore wind farm biological monitoring method, which is applied to a wind farm sea area, wherein a plurality of node areas are arranged in the wind farm sea area, and each node area is internally provided with an information acquisition equipment group and a biological identification terminal, wherein the method comprises the following steps of S101-S103:
step S101, uploading biological information obtained by a biological recognition end for biological recognition in each node area to a service terminal.
Because a plurality of acquisition equipment groups used for biological information acquisition are arranged in each node area, and each acquisition equipment group is in wireless communication with the corresponding biological identification terminal, the real-time monitoring of the biological information in the corresponding area is realized. It can be understood that, in order to improve the reliability of the biological information monitoring, the acquisition device groups corresponding to two adjacent biological identification terminals can communicate with other biological identification terminals adjacent to the biological identification terminal besides the corresponding identification terminal, so as to avoid the influence on the timeliness and reliability of the biological information acquisition caused by the fault or the overlarge load of the current biological identification terminal.
Specifically, the biological information acquisition equipment set comprises a video acquisition equipment set and an audio acquisition equipment set. The video acquisition equipment set comprises a plurality of first video acquisition equipment arranged above water and a plurality of second video acquisition equipment correspondingly arranged below water, and the first video acquisition equipment is associated with the second video acquisition equipment. The audio acquisition equipment group comprises a plurality of hydrophone arrays which are arranged corresponding to the second video acquisition equipment, and the hydrophone arrays are associated with the second video acquisition equipment. It can be understood that each information acquisition device has a unique identification ID and a unique type ID, so that the biological identification terminal can determine the specific installation position of each acquisition device and the type of the acquired information through the identification ID and the type ID.
Further, the first video collecting device and the second video collecting device are respectively a spherical camera and an underwater camera which can rotate 360 degrees. Therefore, video streams of aquatic organisms, ships and underwater organisms in the sea area of the wind power plant can be obtained, and training, verification and testing samples are provided for follow-up biological detection and identification processing. In order to improve the reliability of acquiring images of organisms and ships, the longitude and latitude of the first video acquisition equipment and the longitude and latitude of the second video acquisition equipment are the same, and the first video acquisition equipment and the second video acquisition equipment are provided with associated positioning sensors, deviation sensors and corresponding associated codes, so that the two video acquisition equipment can simultaneously acquire images of the same organism on water, the images of marine organisms can be completely captured, and the reliability of acquiring, identifying and training the images of the organisms can be improved.
Furthermore, the underwater acoustic array sensors are multiple in number, are arranged corresponding to the video acquisition equipment, namely are associated with the video acquisition equipment group, are provided with corresponding associated numbers, locators and deviation sensors, can acquire audio streams, ship noises and other ocean background noises generated by organisms in the sea area of the wind power plant in real time, and provide training, verification and test samples for subsequent biological sound detection, identification and processing. The underwater acoustic array sensor can be configured with corresponding sampling frequency, sampling interval and frequency according to different detection objects, can be arranged on underwater video acquisition equipment, can be arranged on the sea bottom or in sea areas with different depths in a submerged target mode, and can be arranged in a ship dragging mode without limitation.
As can be understood, the accuracy and the reliability of biological information acquisition are improved by dividing the sea area of the wind power plant into a plurality of node areas; the biological information obtained by the biological recognition end which performs biological recognition in each node area is uploaded to the service terminal, so that the service terminal can conveniently enhance the biological information according to the biological information distribution of each biological recognition terminal.
And S102, the server side issues data enhancement information according to the biological information distribution of each biological identification terminal, adjusts the node area according to the biological information distribution after the data enhancement, establishes the node server side according to the adjusted node area, and issues an initial biological identification model to the corresponding biological identification terminal through the node server side.
As described above, due to the uncertainty of the biological activity area in the wind power sea area, the biological identification terminals in each node area have imbalance in biological information acquisition. In order to improve the reliability of model training, data enhancement needs to be performed on the biological information of each biological recognition end. The node area is adjusted according to the biological information distribution of each biological identification end after data enhancement, so that biological information monitoring is realized according to the activity area and the category of the organism. And establishing a node server according to the adjusted node area, and issuing the initial biological recognition model to the corresponding biological recognition terminal through the node server, so as to realize the training of the corresponding biological recognition terminal in each node server on the initial biological recognition model. It can be understood that, in other embodiments, in order to improve the reliability of biometric identification, the number of the information acquisition devices, the position of the wind turbine, and the navigation route of the ship may be adjusted according to the convergence trend graph of the living beings in each node area.
Further, the step S20 further includes:
step S1021, the server receives the biological information of the current node area monitored by the information acquisition equipment group, wherein the biological information comprises a video stream and an audio stream.
Step S1022, performing classification labeling on the image frames and the audio frames in the video stream and the audio stream, respectively, to obtain a biological and marine image data set and a biological and background sound data set.
It can be understood that, by labeling the image frames in the biological information so as to classify and label the organisms and the ships in the image frames, necessary conditions are provided for subsequent identification of the types of the organisms and identification of the ships. By marking the audio frames in the biological information, the biological sound, the ship noise and the ocean background in the audio frames are conveniently classified and marked, and necessary conditions are provided for the identification of subsequent biological types and the identification of ship sounds. The invention provides necessary conditions for the generation and training of the biology and ship image recognition model and the biology and background sound recognition model through the generation of the biology and ship image data set and the biology and background sound data set.
It should be noted that the biological and background sound data sets are obtained by performing spectrum analysis on the collected audio stream, including the start frequency, the end frequency, the minimum frequency, the maximum frequency, the frequency variation, the period, and the like. Spectral feature analysis is carried out on the obtained wind power plant biology and background sound, and the separated single signal sequence is extracted through an energy detection script of MATLAB to serve as a biology and background sound data set.
Further, the method of initial biometric model generation includes:
according to a convolutional neural network and a first activation function corresponding to the neural network, carrying out forward propagation on data in a biological and ship image data set which is input into the convolutional neural network in batches, so as to obtain a biological and ship prediction map; and calculating a first difference value between the biological and ship prediction images and the standard biological and ship images according to a first loss function, carrying out reverse propagation on the first difference value, and obtaining a biological and ship image recognition model through iterative training of a preset period. Wherein the first activation function is a ReLU activation function, and the first loss function is a weighted multi-class cross entropy loss function. It can be understood that the reliability of identifying the living beings and the ships in the wind power sea area is improved by generating the living beings and the ship image identification model.
According to the BP neural network and a second activation function corresponding to the neural network, carrying out forward propagation on data in a biological and background sound data set which are input into the BP neural network in divided batches, so as to obtain a biological and background sound prediction frame; and calculating a second difference value between the biology and background sound prediction frame and the standard biology and background sound frame according to a second loss function, carrying out back propagation on the second difference value, and obtaining a biology and background sound recognition model through iterative training of a preset period. Wherein the second activation function is a PReLU activation function, and the second loss function is a weighted multi-class cross entropy loss function. It can be understood that the reliability of identifying the living beings and the ships in the wind power sea area is improved by generating the living beings and the background sound identification model.
Further, the biological and ship image recognition models and the biological and background sound recognition models are integrated to obtain an initial biological recognition model. Specifically, the initial biological recognition model is obtained by performing data association on the biological and ship image recognition model and the biological and background sound recognition model. The video result and the acoustic result are synchronously analyzed, the relation between the video image data and the acoustic data is established, the underwater fish is monitored together, and the recognition rate is improved. The method has the advantages that through the establishment of the biological recognition model, the reliability of biological and ship recognition is improved, the reduction of the reliability of biological recognition and monitoring of the wind power plant due to the interference of ship images and sounds is avoided, and the key information of the influence of the ship on the biological of the wind power plant can be obtained.
It should be further noted that, before the step of issuing the initial biometric model to the corresponding biometric server through the node server, the method further includes: and the service terminal trains the initial biological recognition model according to the loss function and the biological information after data enhancement until the loss value of the model tends to converge. Wherein the Loss function Center Loss function.
And S103, each node server performs iterative training on each initial biological recognition model through a gradient descent method, uploads the trained initial biological recognition models to the server, and the server performs model fusion on the trained initial biological recognition models through a fusion algorithm to obtain a target biological recognition model.
Specifically, the model is continuously updated according to the difference between the parameters of the biometric identification model trained in the previous round and the learning rate of the biometric identification model and the derivative of the loss function of the biometric identification model trained in the previous round at each biometric identification end, and finally each node server uploads the initial biometric identification model trained by each node server to the server. After receiving an initial biological recognition model obtained by iterative training of each node server, the server side; and taking each trained initial biological recognition model as input, and integrating each trained initial biological recognition model through a federal average algorithm to obtain a target biological recognition model.
It should be further noted that, in other embodiments of the present invention, the method further includes: and issuing the target biological identification model to each biological identification terminal for biological identification, performing static security analysis according to an identification result, and adding biological information of which the identification accuracy is smaller than a standard value into biological identification constraint so as to constrain the target biological model. It can be understood that, in order to improve the reliability of the system biometric identification, the target biometric model needs to be tested and verified, and the model is modified and adjusted according to the verification result.
In specific implementation, each biological recognition end branches video information and audio information acquired by the information acquisition equipment group according to the received target biological model to obtain effective video and audio frames; and determining the type and the quantity of the monitored biological information and the quantity and the type of the ships by using the video frame and the audio frame which have information correlation. The method can be used for carrying out relevance identification on the images of aquatic and underwater organisms in the sea area of the wind power plant, and can be combined with the graphic information direction according to the underwater acoustic array, so that the accuracy and reliability of organism identification are improved, and the condition that the same organism generates sounds with different frequencies due to special reasons to cause identification errors is avoided. In addition, necessary conditions are provided for the separation of target signals through the recognition and training of ship images, ship sounds and background sounds.
According to the method for monitoring the biology of the offshore wind farm, the biological information obtained by a biological identification end for carrying out biological identification in each node area can be uploaded to a service terminal; the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side; each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model. The initial biological recognition models corresponding to the node service terminals are recognized and trained through biological information in each node area in the wind power sea area, and model fusion is carried out on the trained initial biological recognition models through a fusion algorithm, so that the obtained biological recognition models have high robustness, and the reliability monitoring on the biological species distribution, abundance and habitat in the wind power sea area is realized.
It should be understood that, although the steps in the above-described flowcharts are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in the above-described flowcharts may include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of performing the sub-steps or the stages is not necessarily sequential, but may be performed alternately or alternatingly with other steps or at least a portion of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 3, an offshore wind farm biological monitoring system is provided, which is applied to a wind farm sea area, a plurality of node areas are arranged in the wind farm sea area, wherein each node area is provided with an information acquisition device group and a biological identification terminal, and the system includes: an information acquisition module 10, a node creation module 20 and a model generation module 30.
Further, the information obtaining module 10 is configured to upload biological information obtained by a biological recognition end performing biological recognition in each node area to the service terminal.
The biological information acquisition equipment group comprises a video acquisition equipment group and an audio acquisition equipment group.
Specifically, the video acquisition equipment group comprises a plurality of first video acquisition equipment arranged above water and a plurality of second video acquisition equipment correspondingly arranged below water, and the first video acquisition equipment is associated with the second video acquisition equipment.
Specifically, the audio acquisition device group includes a plurality of hydrophone arrays corresponding to the second video acquisition device, and the hydrophone arrays are associated with the second video acquisition device.
Further, the node creating module 20 is configured to enable the server to issue data enhancement information according to the biological information distribution of each biological identification terminal, adjust the node area according to the data enhanced biological information distribution, create a node server according to the adjusted node area, and issue the initial biological identification model to the corresponding biological identification terminal through the node server.
Referring to fig. 4, the node creating module 20 includes:
the information receiving unit 21 is configured to receive biological information of the current node area monitored by the information acquisition device group, where the biological information includes a video stream and an audio stream.
And the classification marking unit 22 is configured to perform classification marking on the image frames and the audio frames in the video stream and the audio stream, respectively, so as to obtain a biological and marine image data set and a biological and background sound data set.
The first training unit 23 is configured to forward propagate data in a biological and marine image data set input to the convolutional neural network in batches according to the convolutional neural network and a first activation function corresponding to the neural network, obtain a biological and marine prediction image, calculate a first difference between the biological and marine prediction image and a standard biological and marine image according to a first loss function, perform back propagation on the first difference, and obtain a biological and marine image recognition model through iterative training of a preset period.
The second training unit 24 is configured to forward propagate data in the biological and background sound data sets input to the BP neural network in a split-batch manner according to the BP neural network and a second activation function corresponding to the BP neural network to obtain a biological and background sound prediction frame, calculate a second difference between the biological and background sound prediction frame and a standard biological and background sound frame according to a second loss function, perform back propagation on the second difference, and obtain a biological and background sound identification model through iterative training in a preset period.
And a model integration unit 25, configured to integrate the biological and marine image recognition models and the biological and background sound recognition models to obtain an initial biological recognition model.
And a third training unit 26, configured to enable the service terminal to train the initial biometric model according to the loss function and the data-enhanced biometric information until the loss value of the model tends to converge.
Further, the model generating module 30 is configured to enable each node server to perform iterative training on each initial biometric model through a gradient descent method, and upload the trained initial biometric models to the server, where the server performs model fusion on each trained initial biometric model through a fusion algorithm to obtain a target biometric model.
Specifically, receiving an initial biological recognition model obtained after iterative training of each node server; and taking each trained initial biological recognition model as input, and integrating each trained initial biological recognition model through a federal average algorithm to obtain a target biological recognition model.
Further, the model generating module 30 is further configured to issue the target biometric model to each biometric terminal for biometric identification, perform static security analysis according to the identification result, and add the biometric information with the identification accuracy rate smaller than the standard value to the constraints of biometric identification, so as to constrain the target biometric model.
For specific limitations of the offshore wind farm biological monitoring system, reference may be made to the above limitations of the offshore wind farm biological monitoring method, which are not described herein again. All or part of the modules in the biological monitoring system of the offshore wind farm can be realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
The biological monitoring system of the offshore wind farm can upload biological information obtained by a biological recognition end for biological recognition in each node area to a service terminal; the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side; each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model. The initial biological recognition models corresponding to the node service terminals are recognized and trained through biological information in each node area in the wind power sea area, and model fusion is carried out on the trained initial biological recognition models through a fusion algorithm, so that the obtained biological recognition models have high robustness, and the reliability monitoring on the biological species distribution, abundance and habitat in the wind power sea area is realized.
In one embodiment, a computer device is provided, which may be a server, the internal structure of which may be as shown in fig. 5. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of terminal application interaction.
Those skilled in the art will appreciate that the architecture shown in fig. 5 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program:
receiving live pig information of a current breeding unit monitored by the information acquisition equipment, and determining an abnormal position bar in the current breeding unit according to the live pig information;
generating group transfer path information according to the position information of the abnormal position bar, the group transfer entrance position and the pig farm electronic map;
and acquiring the position information of the guiding equipment in the farm, and controlling the guiding equipment to switch to the corresponding group turning direction according to the group turning path information.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
receiving live pig information of a current breeding unit monitored by the information acquisition equipment, and determining an abnormal position bar in the current breeding unit according to the live pig information;
generating group transfer path information according to the position information of the abnormal position bar, the group transfer entrance position and the pig farm electronic map;
and acquiring the position information of the guiding equipment in the farm, and controlling the guiding equipment to switch to the corresponding group turning direction according to the group turning path information.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It should be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the various steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any one or combination of the following techniques, which are known in the art, may be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application specific integrated circuit having an appropriate combinational logic gate circuit, a Programmable Gate Array (PGA), a Field Programmable Gate Array (FPGA), or the like.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.

Claims (10)

1. A biological monitoring method for an offshore wind farm is applied to a wind farm sea area, a plurality of node areas are arranged in the wind farm sea area, and the biological monitoring method is characterized in that an information acquisition equipment group and a biological identification terminal are arranged in each node area, and the method comprises the following steps:
uploading biological information obtained by a biological identification end for biological identification in each node area to a service terminal;
the server side issues data enhancement information according to the biological information distribution of each biological recognition end, adjusts the node area according to the biological information distribution after the data enhancement, creates a node server side according to the adjusted node area, and issues an initial biological recognition model to the corresponding biological recognition end through the node server side;
each node server side conducts iterative training on each initial biological recognition model through a gradient descent method, the trained initial biological recognition models are uploaded to the server side, and the server side conducts model fusion on each trained initial biological recognition model through a fusion algorithm to obtain a target biological recognition model.
2. The offshore wind farm biological monitoring method according to claim 1, wherein before the step of issuing the initial biological recognition model to the corresponding biological recognition end through the node server, the method further comprises: and the service terminal trains the initial biological recognition model according to the loss function and the biological information after data enhancement until the loss value of the model tends to converge.
3. The offshore wind farm biological monitoring method of claim 1, further comprising;
receiving biological information of a current node area monitored by the information acquisition equipment group, wherein the biological information comprises a video stream and an audio stream;
and respectively carrying out classification marking on the image frames and the audio frames in the video stream and the audio stream to obtain a biological and ship image data set and a biological and background sound data set.
4. The offshore wind farm biological monitoring method according to claim 3, further comprising:
according to a convolutional neural network and a first activation function corresponding to the neural network, carrying out forward propagation on data in a biological and ship image data set which is input into the convolutional neural network in a divided mode to obtain a biological and ship prediction image, calculating a first difference value between the biological and ship prediction image and a standard biological and ship image according to a first loss function, carrying out backward propagation on the first difference value, and carrying out iterative training in a preset period to obtain a biological and ship image identification model;
according to a BP neural network and a second activation function corresponding to the neural network, carrying out forward propagation on data in biological and background sound data sets input into the BP neural network in batches to obtain a biological and background sound prediction frame, calculating a second difference value between the biological and background sound prediction frame and a standard biological and background sound frame according to a second loss function, carrying out backward propagation on the second difference value, and carrying out iterative training in a preset period to obtain a biological and background sound recognition model;
and integrating the biological and ship image recognition model and the biological and background sound recognition model to obtain an initial biological recognition model.
5. The offshore wind farm biological monitoring method according to claim 1, wherein the method for performing model fusion on each trained initial biological recognition model through a fusion algorithm by the server comprises the following steps:
receiving an initial biological recognition model obtained after iterative training of each node server;
and taking each trained initial biological recognition model as input, and integrating each trained initial biological recognition model through a federal average algorithm to obtain a target biological recognition model.
6. The offshore wind farm biological monitoring method of claim 1, further comprising: and issuing the target biological identification model to each biological identification terminal for biological identification, performing static security analysis according to an identification result, and adding biological information of which the identification accuracy is smaller than a standard value into biological identification constraint so as to constrain the target biological model.
7. The offshore wind farm biological monitoring method according to any one of claims 1 to 6, wherein the biological information collection device group comprises a video collection device group and an audio collection device group,
the video acquisition equipment set comprises a plurality of first video acquisition equipment arranged above water and a plurality of second video acquisition equipment correspondingly arranged below water, and the first video acquisition equipment is associated with the second video acquisition equipment;
the audio acquisition equipment group comprises a plurality of hydrophone arrays of equipment corresponding to the second video acquisition equipment, and the hydrophone arrays are associated with the second video acquisition equipment.
8. The utility model provides an offshore wind farm biological monitoring system, is applied to in a wind-powered electricity generation field sea area, be equipped with a plurality of node areas in the wind-powered electricity generation field sea area, its characterized in that all is equipped with information acquisition equipment group and biological identification terminal in every node area, the system includes:
the information acquisition module is used for uploading biological information acquired by a biological identification terminal for biological identification in each node area to the service terminal;
the node creating module is used for enabling the server to issue data enhancement information according to the biological information distribution of each biological identification terminal, adjusting the node area according to the biological information distribution after the data enhancement, creating a node server according to the adjusted node area, and issuing an initial biological identification model to the corresponding biological identification terminal through the node server;
and the model generation module is used for enabling each node server to carry out iterative training on each initial biological recognition model through a gradient descent method, uploading the trained initial biological recognition models to the server, and carrying out model fusion on each trained initial biological recognition model through a fusion algorithm by the server so as to obtain a target biological recognition model.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
CN202010946992.2A 2020-09-10 2020-09-10 Offshore wind farm biological monitoring method and system, computer equipment and storage medium Active CN112183587B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010946992.2A CN112183587B (en) 2020-09-10 2020-09-10 Offshore wind farm biological monitoring method and system, computer equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010946992.2A CN112183587B (en) 2020-09-10 2020-09-10 Offshore wind farm biological monitoring method and system, computer equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112183587A true CN112183587A (en) 2021-01-05
CN112183587B CN112183587B (en) 2022-04-08

Family

ID=73921716

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010946992.2A Active CN112183587B (en) 2020-09-10 2020-09-10 Offshore wind farm biological monitoring method and system, computer equipment and storage medium

Country Status (1)

Country Link
CN (1) CN112183587B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113250134A (en) * 2021-04-28 2021-08-13 华能澜沧江水电股份有限公司 Fish habitat protection system and construction method thereof
CN113554123A (en) * 2021-09-18 2021-10-26 江苏禹治流域管理技术研究院有限公司 Automatic sand production ship identification method based on acousto-optic linkage
CN114689534A (en) * 2022-04-01 2022-07-01 北京千尧新能源科技开发有限公司 Marine microorganism monitoring method and related equipment
CN117078116A (en) * 2023-10-17 2023-11-17 华能(浙江)能源开发有限公司清洁能源分公司 Robustness analysis method and system for influence of wind power plant site selection on marine biota

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105785968A (en) * 2014-12-25 2016-07-20 中国科学院声学研究所 Marine ranch acoustic monitoring system and monitoring method
CN108924483A (en) * 2018-06-27 2018-11-30 南京朴厚生态科技有限公司 A kind of automatic monitoring system and method for the field animal based on depth learning technology
CN110929880A (en) * 2019-11-12 2020-03-27 深圳前海微众银行股份有限公司 Method and device for federated learning and computer readable storage medium
CN110956202A (en) * 2019-11-13 2020-04-03 重庆大学 Image training method, system, medium and intelligent device based on distributed learning
CN111611610A (en) * 2020-04-12 2020-09-01 西安电子科技大学 Federal learning information processing method, system, storage medium, program, and terminal

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105785968A (en) * 2014-12-25 2016-07-20 中国科学院声学研究所 Marine ranch acoustic monitoring system and monitoring method
CN108924483A (en) * 2018-06-27 2018-11-30 南京朴厚生态科技有限公司 A kind of automatic monitoring system and method for the field animal based on depth learning technology
CN110929880A (en) * 2019-11-12 2020-03-27 深圳前海微众银行股份有限公司 Method and device for federated learning and computer readable storage medium
CN110956202A (en) * 2019-11-13 2020-04-03 重庆大学 Image training method, system, medium and intelligent device based on distributed learning
CN111611610A (en) * 2020-04-12 2020-09-01 西安电子科技大学 Federal learning information processing method, system, storage medium, program, and terminal

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LI-WEI FONG: "《Multi-sensor Data Fusion via Federated Adaptive Filter》", 《2010 INTERNATIONAL SYMPOSIUM ON COMPUTER, COMMUNICATION, CONTROL AND AUTOMATION》 *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113250134A (en) * 2021-04-28 2021-08-13 华能澜沧江水电股份有限公司 Fish habitat protection system and construction method thereof
CN113554123A (en) * 2021-09-18 2021-10-26 江苏禹治流域管理技术研究院有限公司 Automatic sand production ship identification method based on acousto-optic linkage
CN113554123B (en) * 2021-09-18 2022-01-21 江苏禹治流域管理技术研究院有限公司 Automatic sand production ship identification method based on acousto-optic linkage
CN114689534A (en) * 2022-04-01 2022-07-01 北京千尧新能源科技开发有限公司 Marine microorganism monitoring method and related equipment
CN114689534B (en) * 2022-04-01 2023-01-31 北京千尧新能源科技开发有限公司 Marine microorganism monitoring method and related equipment
CN117078116A (en) * 2023-10-17 2023-11-17 华能(浙江)能源开发有限公司清洁能源分公司 Robustness analysis method and system for influence of wind power plant site selection on marine biota
CN117078116B (en) * 2023-10-17 2024-02-27 华能(浙江)能源开发有限公司清洁能源分公司 Robustness analysis method and system for influence of wind power plant site selection on marine biota

Also Published As

Publication number Publication date
CN112183587B (en) 2022-04-08

Similar Documents

Publication Publication Date Title
CN112183587B (en) Offshore wind farm biological monitoring method and system, computer equipment and storage medium
CN114362367B (en) Cloud-edge-cooperation-oriented power transmission line monitoring system and method, and cloud-edge-cooperation-oriented power transmission line identification system and method
US12078775B2 (en) Ocean weather forecasting system
CN110874550A (en) Data processing method, device, equipment and system
CN112183166B (en) Method and device for determining training samples and electronic equipment
Sueur et al. Global estimation of animal diversity using automatic acoustic sensors
Zualkernan et al. An aiot system for bat species classification
CN109473119B (en) Acoustic target event monitoring method
Gallacher et al. Shazam for bats: Internet of Things for continuous real‐time biodiversity monitoring
WO2019238232A1 (en) Method and machine readable storage medium of classifying a near sun sky image
CN108810551B (en) Video frame prediction method, terminal and computer storage medium
CN111967419B (en) Dam bank dangerous case prediction method, dam bank dangerous case prediction device, computer equipment and storage medium
Bosma et al. Estimating solar and wind power production using computer vision deep learning techniques on weather maps
CN115630326B (en) Method and device for monitoring health state of marine ecosystem by hydrophone
CN116170874A (en) Robust WiFi fingerprint indoor positioning method and system
Poupard et al. From massive detections and localisations of orca at orcalab over three years to real-time survey joint to environmental conditions
Constantinou et al. Modeling the operating characteristics of iot for underwater sound classification
Zuba et al. UPAL: Underwater passive aquatic listener
Çoban et al. Towards High Resolution Weather Monitoring With Sound Data
Heath The Assessment and Development of Methods in (Spatial) Sound Ecology
CN113033414A (en) Power consumption data anomaly detection method and device, computer equipment and storage medium
CN114153889A (en) Power station monitoring system and method, readable medium and electronic device
Suzuki et al. Visualizing Directional Soundscapes of Bird Vocalizations Using Robot Audition Techniques
Rusu et al. On the design of an acoustic based wildlife intruder detection system
RU150422U1 (en) MOBILE HYDROACOUSTIC INFORMATION COLLECTION, PROCESSING AND OPERATIONAL STORAGE SYSTEM

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant