CN110866593B - Highway severe weather identification method based on artificial intelligence - Google Patents
Highway severe weather identification method based on artificial intelligence Download PDFInfo
- Publication number
- CN110866593B CN110866593B CN201911068506.5A CN201911068506A CN110866593B CN 110866593 B CN110866593 B CN 110866593B CN 201911068506 A CN201911068506 A CN 201911068506A CN 110866593 B CN110866593 B CN 110866593B
- Authority
- CN
- China
- Prior art keywords
- weather
- feature
- severe
- expressway
- classification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an artificial intelligence-based method for identifying severe weather of an expressway, which is used for automatically identifying various weather conditions of the expressway, such as sunny days, rainy days, foggy days (little fog), foggy days (big fog), snow cover and the like. The method comprises the steps of constructing a severe weather data set of the expressway, extracting weather visual features, partitioning a weather feature map, intensively classifying weather, fusing weather results and finally obtaining a weather classification result. On the basis of the traditional deep learning-based classification algorithm, the invention provides a convolutional neural network for classifying the weather of the highway by using local characteristic information; meanwhile, a corresponding network training method is designed aiming at the special structure of the network, and the network can better pay attention to hash weather elements such as raindrops, snow and the like distributed in the monitoring video frame by utilizing the local characteristic information, so that the classification accuracy is improved.
Description
Technical Field
The invention relates to the technical field of intelligent monitoring, in particular to an artificial intelligence-based expressway severe weather identification method.
Background
Poor natural weather conditions such as fog, rainy days and snow often cause the problems of low road visibility, wet and slippery road surfaces and the like, so that serious traffic accidents are caused, the normal use of the expressway is greatly influenced, and the driving safety of vehicles on the expressway is challenged. Therefore, the traffic management department often needs to know the weather conditions of each road section in real time so as to manage and control the traffic state of the expressway and avoid serious traffic accidents.
The network-connected image acquisition cameras are installed on all road sections of the existing expressway, managers can monitor the road weather conditions of all road sections in a data center in real time, and once the condition that severe weather conditions exist in the road sections is found, relevant departments can be informed to timely conduct traffic control at the first time. However, the criss-cross highway generates monitoring video streams of thousands of road sections, and the traffic management department needs to invest a large amount of human resources to monitor each road section in real time. Therefore, an intelligent recognition algorithm capable of automatically processing the monitoring video stream and analyzing the road weather condition is urgently needed.
The traditional visual classification algorithm based on deep learning usually needs classified objects to occupy a high proportion of images and have obvious visual features. However, elements such as raindrops and snow cover which appear in the task of identifying severe weather often occupy a low proportion in the monitoring video frame, the visual characteristics are not obvious, and the scene of the monitoring video generates great difference along with the change of the road section. This makes it difficult for conventional classification algorithms based on deep learning to better identify the weather conditions of the highway.
Disclosure of Invention
In view of the above problems, an object of the present invention is to provide an artificial intelligence-based method for identifying severe weather in an expressway, wherein a convolutional neural network for classifying weather in the expressway is implemented by using local feature information based on a traditional deep learning-based classification algorithm, and the network can better pay attention to hash weather elements such as raindrops, snow and the like distributed in a surveillance video frame by using the local feature information, thereby improving classification accuracy. The technical scheme is as follows:
an artificial intelligence-based method for identifying severe weather of a highway comprises the following steps:
step 1: constructing a severe weather data set of the expressway: acquiring video data of each road section under different weather conditions through a monitoring camera arranged on the highway, and arranging to form a highway severe weather data set comprising different severe weather image data and highway scene image data;
step 2: extracting weather visual features: training a convolutional neural network by using a highway severe weather data set, extracting visual characteristic information of key frames extracted from monitoring video frames to obtain a weather characteristic diagram, so that the weather characteristic key information in the original video frames is obvious, and simultaneously inhibiting the characteristics of background information to reduce the interference of the background information on a severe weather classifier;
and step 3: partitioning a weather characteristic diagram: uniformly blocking the acquired weather feature map to obtain a weather feature sub-map corresponding to a certain local area in the original video frame;
and 4, step 4: and (3) dense weather classification: classifying severe weather of each weather feature sub-graph respectively to obtain the probability of various weather in a pixel area in an original video frame corresponding to each weather feature sub-graph;
and 5: and (3) weather result fusion: and combining probability results corresponding to different weather feature sub-graphs in the same key frame, and synthesizing result information of different areas in the key frame to obtain the probability that the current key frame contains a certain severe weather so as to obtain a weather classification result.
Further, the weather feature map blocks specifically include:
step 31: carrying out dimensionality expansion on a four-dimensional characteristic diagram matrix of a video frame characteristic diagram including a batch, a width, a height and a channel to obtain a five-dimensional characteristic diagram matrix which is convenient for blocking and includes the batch, the 1, the width, the height and the channel;
step 32: and performing linear transformation on the obtained five-dimensional feature map matrix, and uniformly blocking the key frame feature map according to the vector of each position to obtain feature subgraphs, namely a new feature matrix comprising batches, blocks, subgraph width, subgraph height and channels, wherein each feature subgraph corresponds to the weather depth visual features of a specific pixel region in the original video frame.
Further, the weather intensive classification is specifically:
step 41: collecting channel information of the weather feature subgraphs and classifying the channel information to obtain classification result features through a weather classifier formed by two fully-connected layers;
step 42: and (4) passing the classification result features through a specific activation function one by one to obtain the probability that the corresponding weather feature subgraphs belong to various severe weathers.
Further, the weather result fusion specifically comprises:
step 51: adding the probabilities of each weather feature sub-graph in the same key frame belonging to a certain day to average, and taking the probabilities as the probabilities of the current video frame belonging to the specific category to finally obtain the probability vector of the weather classification of the key frame;
step 52: and selecting the element with the maximum probability value from the probability vector of the same key frame, and taking the weather category corresponding to the element as the weather classification result of the current key frame.
The invention has the beneficial effects that: on the basis of the traditional deep learning-based classification algorithm, the invention provides a convolutional neural network for classifying the weather of the highway by using local characteristic information; meanwhile, a corresponding network training method is designed aiming at the special structure of the network, and the network can better pay attention to hashing weather elements such as raindrops, snow and the like distributed in a monitoring video frame by utilizing local characteristic information, so that the classification accuracy is improved.
Drawings
FIG. 1 is a flow chart of the steps of the artificial intelligence-based method for identifying severe weather on a highway.
Fig. 2 is a network structure diagram of the artificial intelligence-based method for identifying severe weather on a highway according to the present invention.
FIG. 3 is a block structure diagram of the severe weather identification method for the expressway according to the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the specific embodiments. The embodiment of the invention relates to an artificial intelligence-based expressway severe weather identification method, which comprises weather visual feature extraction, weather feature map partitioning, weather feature dense classification and weather result fusion, wherein the specific algorithm processing flow is shown in figure 1, and the network structure is shown in figure 2:
step 1: the monitoring cameras installed on the expressway are used for collecting data of all road sections under different weather conditions and arranging and forming the severe weather data sets required by the method, wherein the severe weather data sets comprise image data of sunny days, rainy days, foggy days (little fog), foggy days (heavy fog), various kinds of snow weathers and expressway scenes, so that the model can learn different types of weathers more uniformly and has good robustness.
Step 2: extracting weather visual features: the method comprises the following steps of training a convolutional neural network to extract visual characteristic information of a monitoring video frame by using a severe weather classification data set of the expressway, acquiring a weather characteristic diagram, enabling weather characteristic key information in original video frames such as fog blocks, raindrops and snow to be more remarkable in the characteristic diagram, and simultaneously inhibiting the characteristics of background information such as road scenes and vehicles so as to reduce the interference of the information on a severe weather classifier, wherein the specific steps comprise:
A. for the transmitted highway monitoring video stream, 10 frames are extracted every second to serve as classification key frames, the long sides and the short sides of the key frames are changed into the same size and are scaled to 299 pixel values, and the scaled video frames are further put into a visual feature extraction network by taking every 10 frames of the key frames as a batch;
B. an inclusion v3 model trained on a severe weather classification dataset of a highway is used as a backbone network for visual feature extraction, the key video frame processed in the A is used as input, and the depth visual feature corresponding to the size of (8, 8) is output. Aiming at the characteristic that fog blocks, raindrops and snow cover occupy a small area in a scene in severe weather data, the original inclusion v3 network structure is modified, the last average pooling layer is removed, so that the visual features extracted by the network do not lose local information, and the network structure is shown in a graph II.
And 3, step 3: partitioning a weather characteristic diagram: and (3) uniformly partitioning the weather feature map extracted by using the convolutional neural network in the step (2) to obtain a weather feature sub-map, wherein the weather feature sub-map corresponds to a certain local area in the original video frame, so that the features of key local information such as fog blocks, raindrops, snow cover and the like which are dispersed or have small proportion in the original video frame can occupy a higher proportion in the feature sub-map. The method comprises the following specific steps:
A. acquiring a result of the feature extraction in the step 2, and performing dimension expansion on the video frame feature map including the four-dimensional feature map matrix of the batch, width, height and channel to obtain a five-dimensional feature map matrix which is convenient for blocking and includes the batch, 1, width, height and channel; for example, dimension expansion is performed on the key frame feature map matrix with the matrix size of (10, 8, 8, 2048) to obtain a new matrix with the size of (10, 1, 8, 8, 2048).
B. And performing linear transformation on the obtained five-dimensional feature map matrix, uniformly blocking the key frame feature map according to the vector of each position to obtain 64 feature sub-maps with the size of (1,1,2048), and enabling the size of the formed new feature matrix to be (10, 64, 1,1,2048), namely the new feature matrix comprising batch, blocks, sub-map width, sub-map height and channels, as shown in fig. 3. Each feature sub-image corresponds to the depth visual features of a (37, 37) pixel region in the original video frame, so that the visual features of sporadically distributed weather elements such as rain, snow, fog and the like in the feature sub-images are more obvious.
And 4, step 4: and (3) dense weather classification: and (3) classifying the severe weather of each weather feature sub-image obtained by blocking in the step (3) respectively to obtain the probability of the weather such as fog, rain, snow, sunny and the like in the pixel region in the original video frame corresponding to each feature sub-image of the weather, so that the partial detail information which is not significant in the original video frame can be fully utilized. The method comprises the following specific steps:
A. and (4) receiving the feature matrix obtained in the step (3), and summarizing channel information of the weather feature subgraphs for classification through a weather classifier formed by two fully-connected layers. The number of input channels of the first layer of full-connection structure is 2048, the number of output channels of the first layer of full-connection structure is 1024, and the number of input channels of the second layer of full-connection structure is 1024 and the number of output channels of the second layer of full-connection structure is 4.
B. And receiving the obtained classification result characteristics of each weather characteristic subgraph, and enabling the classification result characteristics to pass through a softmax activation function one by one to obtain the probability that the corresponding weather characteristic subgraph belongs to five types of severe weather, namely sunny weather, rainy weather, foggy weather (small fog), foggy weather (large fog) and snow cover.
And 5: and (3) weather result fusion: and 4, combining the probability results corresponding to different weather feature sub-graphs in the same key frame obtained in the step 4, and synthesizing the result information of different areas in the key frame to obtain the probability that the current key frame contains a certain severe weather. The method comprises the following specific steps:
A. receiving the weather feature sub-graph classification result obtained in the step 4, adding the probabilities of each weather feature sub-graph in the same key frame belonging to a certain day to average, taking the probabilities as the probabilities of the current video frame belonging to the specific category, and finally obtaining the probability vector P ═ P (P) of the weather classification of the key frame 1 ,p 2 ,p 3 ,p 4 ,p 5 ) The probabilities of the current key frame belonging to five types of severe weather, namely sunny weather, rainy weather, foggy weather (little fog), foggy weather (heavy fog) and snow accumulation are respectively corresponded.
B. Receiving the probability vector P of each key frame, averaging the corresponding probability vectors of 10 key frames per second to obtain the weather probability vector P of the highway section at the current time T The calculation formula is as follows:
wherein, P i A weather probability vector representing the ith keyframe. Finally, from P T And taking the weather category corresponding to the element with the maximum probability value as the weather classification result of the expressway section at the current time.
Claims (3)
1. An artificial intelligence-based highway severe weather identification method is characterized by comprising the following steps:
step 1: constructing a severe weather data set of the expressway: acquiring video data of each road section under different weather conditions through a monitoring camera arranged on the expressway, and sorting to form an expressway severe weather data set comprising different severe weather image data and expressway scene image data;
step 2: extracting weather visual features: training a convolutional neural network by using a highway severe weather data set, extracting visual characteristic information of key frames extracted from monitoring video frames to obtain a weather characteristic diagram, so that the weather characteristic key information in original video frames is obvious, and simultaneously inhibiting the characteristics of background information to reduce the interference of the background information on a severe weather classifier;
and step 3: partitioning a weather characteristic diagram: uniformly partitioning the acquired weather feature map to obtain a weather feature sub-map corresponding to a local area in the original video frame, wherein the partitioning of the weather feature map specifically comprises the following steps:
step 31: carrying out dimension expansion on a four-dimensional characteristic diagram matrix of a video frame characteristic diagram comprising a batch, a width, a height and a channel to obtain a five-dimensional characteristic diagram matrix which is convenient for blocking and comprises the batch, the 1, the width, the height and the channel;
step 32: performing linear transformation on the obtained five-dimensional feature map matrix, and uniformly blocking the key frame feature map according to the vector of each position to obtain feature subgraphs, namely a new feature matrix comprising batches, blocks, subgraph widths, subgraph heights and channels, wherein each feature subgraph corresponds to the weather depth visual feature of a specific pixel region in the original video frame
And 4, step 4: and (3) dense weather classification: classifying severe weather of each weather feature sub-graph respectively to obtain the probability of various weather in a pixel area in an original video frame corresponding to each weather feature sub-graph;
and 5: and (3) weather result fusion: and combining probability results corresponding to different weather feature sub-graphs in the same key frame, and synthesizing result information of different areas in the key frame to obtain the probability that the current key frame contains a certain severe weather so as to obtain a weather classification result.
2. The severe weather identification method for the expressway according to claim 1, wherein the weather intensive classification is specifically as follows:
step 41: collecting channel information of the weather feature subgraphs and classifying the channel information to obtain classification result features through a weather classifier formed by two fully-connected layers;
step 42: and (4) enabling the classification result characteristics to pass through a specific activation function one by one to obtain the probability that the corresponding weather characteristic sub-graph belongs to various severe weathers.
3. The severe weather identification method for the expressway according to claim 1, wherein the weather result fusion specifically comprises:
step 51: adding the probabilities of each weather feature sub-graph in the same key frame belonging to a certain day to calculate the average, taking the average as the probability of the current video frame belonging to the weather classification, and finally obtaining the probability vector of the key frame weather classification;
step 52: and selecting the element with the maximum probability value from the probability vector of the same key frame, and taking the weather category corresponding to the element as the weather classification result of the current key frame.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911068506.5A CN110866593B (en) | 2019-11-05 | 2019-11-05 | Highway severe weather identification method based on artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911068506.5A CN110866593B (en) | 2019-11-05 | 2019-11-05 | Highway severe weather identification method based on artificial intelligence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110866593A CN110866593A (en) | 2020-03-06 |
| CN110866593B true CN110866593B (en) | 2022-09-23 |
Family
ID=69653530
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911068506.5A Active CN110866593B (en) | 2019-11-05 | 2019-11-05 | Highway severe weather identification method based on artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110866593B (en) |
Families Citing this family (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111797912B (en) * | 2020-06-23 | 2023-09-22 | 山东浪潮超高清视频产业有限公司 | System and method for identifying film age type and construction method of identification model |
| CN112672209A (en) * | 2020-12-14 | 2021-04-16 | 北京达佳互联信息技术有限公司 | Video editing method and video editing device |
| CN113129273B (en) * | 2021-03-30 | 2023-04-07 | 武汉伏佳安达电气技术有限公司 | Automatic transmission line fog inspection method and system |
| CN113869347B (en) * | 2021-07-20 | 2022-08-02 | 西安理工大学 | Fine-grained classification method for severe weather image |
| CN114120025A (en) * | 2021-09-29 | 2022-03-01 | 吉林大学 | Deep learning-based weather identification and degree quantification method |
| CN113793511B (en) * | 2021-11-18 | 2022-02-15 | 西南交通大学 | Highway vehicle driving indication method, device, equipment and readable storage medium |
| CN114299726B (en) * | 2021-12-31 | 2024-07-19 | 象谱信息产业有限公司 | Expressway bad weather identification method based on artificial intelligence |
| CN115310519A (en) * | 2022-07-11 | 2022-11-08 | 广州国交润万交通信息有限公司 | Method for weather discrimination by combining video and audio |
| CN116092013B (en) * | 2023-03-06 | 2023-06-09 | 广东汇通信息科技股份有限公司 | Dangerous road condition identification method for intelligent monitoring |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722993A (en) * | 2012-06-01 | 2012-10-10 | 上海交通大学 | Highway severe weather monitoring system based on wireless sensing network |
| CN109685146A (en) * | 2018-12-20 | 2019-04-26 | 浙江师范大学 | A kind of scene recognition method based on double convolution sum topic models |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9613269B2 (en) * | 2014-03-31 | 2017-04-04 | Honeywell International Inc. | Identifying and tracking convective weather cells |
| CN104240239B (en) * | 2014-09-01 | 2018-05-01 | 西南交通大学 | A kind of detection method based on road image detection local section haze weather |
| KR101933856B1 (en) * | 2017-07-03 | 2018-12-31 | (주)시정 | System for image processing using the convolutional neural network and method using the same |
| CN111133447B (en) * | 2018-02-18 | 2024-03-19 | 辉达公司 | Method and system for object detection and detection confidence for autonomous driving |
| CN108875593A (en) * | 2018-05-28 | 2018-11-23 | 上海交通大学 | Visible images weather recognition methods based on convolutional neural networks |
| CN109214470B (en) * | 2018-10-25 | 2020-11-03 | 中国人民解放军国防科技大学 | Image visibility detection method based on coding network fine adjustment |
| CN110378865A (en) * | 2019-04-28 | 2019-10-25 | 湖南师范大学 | A kind of greasy weather visibility intelligence hierarchical identification method and system under complex background |
-
2019
- 2019-11-05 CN CN201911068506.5A patent/CN110866593B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102722993A (en) * | 2012-06-01 | 2012-10-10 | 上海交通大学 | Highway severe weather monitoring system based on wireless sensing network |
| CN109685146A (en) * | 2018-12-20 | 2019-04-26 | 浙江师范大学 | A kind of scene recognition method based on double convolution sum topic models |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110866593A (en) | 2020-03-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110866593B (en) | Highway severe weather identification method based on artificial intelligence | |
| CN112101433B (en) | Automatic lane-dividing vehicle counting method based on YOLO V4 and DeepSORT | |
| CN109977812B (en) | Vehicle-mounted video target detection method based on deep learning | |
| US9852344B2 (en) | Systems and methods for semantically classifying and normalizing shots in video | |
| Dornaika et al. | Building detection from orthophotos using a machine learning approach: An empirical study on image segmentation and descriptors | |
| DE112013001858B4 (en) | Multiple-hint object recognition and analysis | |
| CN107563413B (en) | Unmanned aerial vehicle aerial image farmland block object accurate extraction method | |
| US20130259375A1 (en) | Systems and Methods for Semantically Classifying and Extracting Shots in Video | |
| CN104134068B (en) | Monitoring vehicle feature representation and classification method based on sparse coding | |
| CN106682586A (en) | Method for real-time lane line detection based on vision under complex lighting conditions | |
| CN109063667B (en) | Scene-based video identification mode optimization and pushing method | |
| CN104978567A (en) | Vehicle detection method based on scenario classification | |
| CN110866879A (en) | Image rain removing method based on multi-density rain print perception | |
| CN110706239A (en) | Scene segmentation method fusing full convolution neural network and improved ASPP module | |
| CN110163822A (en) | The netted analyte detection and minimizing technology and system cut based on super-pixel segmentation and figure | |
| CN107590486B (en) | Moving object identification method and system, and bicycle flow statistical method and equipment | |
| CN115240024A (en) | Method and system for segmenting extraterrestrial pictures by combining self-supervised learning and semi-supervised learning | |
| Tariq et al. | Real time vehicle detection and colour recognition using tuned features of Faster-RCNN | |
| Ghahremannezhad et al. | A new adaptive bidirectional region-of-interest detection method for intelligent traffic video analysis | |
| JP2011150605A (en) | Image area dividing device, image area dividing method, and image area dividing program | |
| Hänisch et al. | Free-space detection with fish-eye cameras | |
| CN113887532A (en) | Method for identifying and correcting accumulated snow images of expressway based on scene classification | |
| CN110705394B (en) | Scenic spot crowd behavior analysis method based on convolutional neural network | |
| Zhao et al. | Weather classification based on convolutional neural networks | |
| CN107341456B (en) | Weather sunny and cloudy classification method based on single outdoor color image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |