[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN118155161A - Road isolation belt detection method and device based on image global receptive field - Google Patents

Road isolation belt detection method and device based on image global receptive field Download PDF

Info

Publication number
CN118155161A
CN118155161A CN202410105601.2A CN202410105601A CN118155161A CN 118155161 A CN118155161 A CN 118155161A CN 202410105601 A CN202410105601 A CN 202410105601A CN 118155161 A CN118155161 A CN 118155161A
Authority
CN
China
Prior art keywords
image
isolation belt
receptive field
road
road isolation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410105601.2A
Other languages
Chinese (zh)
Inventor
胡莹
黄飞翔
金�一
苏楦雯
王涛
李浥东
王书灵
王伟
白海珍
荆禄波
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Transport Institute
Beijing Jiaotong University
Original Assignee
Beijing Transport Institute
Beijing Jiaotong University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Transport Institute, Beijing Jiaotong University filed Critical Beijing Transport Institute
Priority to CN202410105601.2A priority Critical patent/CN118155161A/en
Publication of CN118155161A publication Critical patent/CN118155161A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/0464Convolutional networks [CNN, ConvNet]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/048Activation functions
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/084Backpropagation, e.g. using gradient descent
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/08Learning methods
    • G06N3/09Supervised learning
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/42Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/44Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • G06V10/443Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
    • G06V10/449Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
    • G06V10/451Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
    • G06V10/454Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/771Feature selection, e.g. selecting representative features from a multi-dimensional feature space
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/776Validation; Performance evaluation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/70Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Biodiversity & Conservation Biology (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a road isolation belt detection method and device based on an image global receptive field, which belong to the technical field of road detection based on machine vision and acquire road isolation belt images to be detected; processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained by training based on the training data; and obtaining a predicted road isolation belt form through a forward propagation process of layer-by-layer convolution, calculating the loss between a predicted result and a real label, and carrying out backward propagation through the loss to update the model weight until the set iteration round number is reached. The invention increases the receptive field of the network layer, accurately judges the mechanical-non-isolation form of the scene, and improves the comprehensiveness and accuracy of the prediction result.

Description

Road isolation belt detection method and device based on image global receptive field
Technical Field
The invention relates to the technical field of road detection based on machine vision, in particular to a road isolation belt detection method and device based on an image global receptive field.
Background
Road barriers are provided between motor vehicles and non-motor vehicles in order to enhance traffic safety and regulate traffic flow. The isolation belt is used as a physical isolator, so that motor vehicles and non-motor vehicles can be effectively divided, cross interference between the motor vehicles and the non-motor vehicles is reduced, and the occurrence probability of traffic accidents is reduced. In addition, the road isolation belt is favorable for maintaining traffic order, promoting traffic smoothness, realizing the diversion of motor vehicles, non-motor vehicles and pedestrians, and forming a highly ordered and safe traffic system. Establishes a solid foundation for the efficient operation of the urban slow-moving traffic system.
The detection of road barriers between motor vehicles and non-motor vehicles is a critical task in the field of slow traffic. Today, road isolation belts come in a variety of forms, including fences, solid lines, green belts, cement, and the like. Accurate identification of the type of roadway barrier is critical to traffic management and urban planning. Different types of roadway barriers may have different effects on traffic flow, while automatic identification may provide valuable information for traffic planning and safety. With the continuous development of intelligent traffic technology, automatic identification of road isolation zones by using image processing and artificial intelligence technology becomes more feasible.
In the field of computer vision, image classification has been a very critical person, whose main objective is to divide an input image into different predefined categories. To achieve high precision image classification, researchers have proposed a number of different approaches, including traditional classical approaches and the latest deep learning techniques. The core idea of these methods is generally to extract features from an image and then use these features to classify to determine the category to which the image belongs.
With the rapid development of deep learning technology, deep neural networks are increasingly used in image classification tasks. In particular Convolutional Neural Networks (CNNs) have become the method of choice for image classification tasks. However, conventional convolutional neural networks may face some challenges for certain specific image classification tasks, such as road-isolation-band-style identification. Because road barriers are of various types, they may have different shapes, colors and textures in different scenes. Conventional CNNs may be limited in handling this diversity, and thus more advanced methods are needed to address this problem.
In this context, a global receptive field-based residual neural network appears, which is a deep learning method, by introducing residual connection, so that the network can learn residual information in images, thereby better processing various complex and diversified image features. Global receptive field means that each neuron in the network has a comprehensive perception and understanding of the overall information of the input image. The expanding of the receptive field enables the residual neural network to better capture the context information and details in the image, thereby improving the accuracy and the robustness of the image classification task.
At present, no practical application scheme similar to the present invention for identifying the form of the road isolation belt has been presented. However, it is worth noting that this task can essentially be seen as converting the detection problem into an image classification problem. In terms of a network structure of an algorithm model, resNet (Residual Network) provides a residual block structure, solves the problem that a network is difficult to train due to gradient disappearance or gradient explosion in a deep neural network, and solves the problem that network performance is lowered due to deeper layers, so that the problem is excellent in an image classification task.
ResNet the network architecture is based on a deep convolutional neural network, but its core innovation is the introduction of a Residual Block (Residual Block) architecture. In this structure, each residual block contains two branches, one is an identity map (IDENTITY MAPPING) and the other is a learned residual. This design enables the network to learn residual information, i.e. differences in the image relative to the identity map, instead of directly learning the complete feature map. This solves the gradient vanishing and gradient explosion problems in conventional deep networks, enabling deeper network structures to be trained and optimized.
In the task of image classification ResNet, by virtue of its depth scalability and high performance, is widely used. By stacking multiple residual blocks, the network can achieve a very deep structure, effectively capturing complex features in the image. In addition, due to the design of the residual block, increasing the network depth does not lead to performance degradation any more, but rather improves classification accuracy.
In summary, in the prior art, a method and a device for automatically detecting a road isolation belt by adopting a deep learning mechanism are lacking, and most of the current methods still rely on manual labeling for classification, so that the method is difficult to meet the requirements in the intelligent traffic background. When the convolutional neural network is directly adopted to classify traffic images, the actual receptive field is usually smaller, and the theoretical receptive field is insufficient to cover the whole road isolation zone, so that incomplete capturing of information can be caused. Furthermore, the downsampling process (e.g., pooling) may result in the loss of a large amount of critical information, whereas in the case of non-isolated band recognition tasks, isolated bands typically span the entire image, thus requiring a larger receptive field to achieve more accurate prediction output.
Disclosure of Invention
The invention aims to provide a road isolation belt detection method and device based on an image global receptive field, which are used for solving at least one technical problem in the background technology.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
on one hand, the invention provides a road isolation belt detection model training method based on an image global receptive field, which comprises the following steps:
Acquiring training data; the training data comprises a plurality of road isolation belt images and labels for labeling isolation belt types in the images;
Training the road isolation belt detection model based on the image global receptive field based on the training data; the method comprises the steps of obtaining a predicted road isolation belt form through a forward propagation process of layer-by-layer convolution, calculating loss between a predicted result and a real label, and carrying out backward propagation through the loss to update model weights until the set iteration number is reached; the road isolation belt detection model based on the image global receptive field comprises a multi-scale enhancement module, wherein the multi-scale enhancement module is used for reducing the number of input channels by a 1x1 convolution layer and batch normalization, a series of 3x3 convolution layers and batch normalization layers are used for feature extraction and multi-scale information, then the 1x1 convolution layers and batch normalization layers are used for increasing the number of output channels, finally a ReLU activation function is used for reducing the size of an input feature map, and an optional downsampling part is used for reducing the size of the input feature map.
Optionally, a network structure of a road isolation belt detection model based on an image global receptive field adopts a ResNet residual network structure, and a group of smaller filter banks are introduced to replace a single 3×3 convolution kernel of n channels; wherein each filter bank contains w channels, n=s×w, s being the number of filters in the filter bank; these filter banks are interrelated in a hierarchical fashion like a residual connection, increasing the dimensional diversity of the output features.
Optionally, for the input features, the number of channels is adjusted using a1×1 convolutional layer; dividing the input features into three levels; each group of filters firstly extracts features from one group of input feature graphs, and the output features of the previous group are transmitted to the next group of filters together with the other group of input feature graphs until all the input feature graphs are processed; all sets of feature maps are concatenated together and input into a1 x 1 convolutional layer to achieve complete information fusion.
Optionally, calculating the loss of each training sample by traversing the training dataset and performing back propagation to update the model parameters; at regular intervals of iterations, the model propagates forward through the test dataset, calculating test loss and accuracy to evaluate the performance of the model on unseen data until a specified number of iterations is reached.
Optionally, acquiring training data includes: the traffic image is divided into different forms of road isolation belts: "./FACILITY STRIP" indicates that the road barrier of the image is cement or green belt, "/fence" indicates that the road barrier of the image is fence, "/line" indicates that the road barrier of the image is solid line and "/else" indicates that the road barrier of the image is in a form other than the above three cases; and reading and processing the data set image by using codes to obtain training data and a test data set.
Optionally, reading and processing the dataset image with the code to obtain training data and test data includes:
Acquiring file paths of all images in a folder containing image files under a code-specified path by using a glob library, a glob. Glob method;
Four image categories are defined: 'else', 'FACILITY STRIP', 'fence', and 'line', and the mapping relationship between them and integer tags;
Creating a custom data set class MYDATASET which inherits the self-supporting image path, the label and the data transformation function in the initialization process, wherein the custom data set class is used for processing image data;
the data transformation function transform defines a series of transformation operations including resizing the image to 256x256 pixels and transforming the image to a sheet;
creating a data loader dataloader for dividing the dataset into batches, each batch containing 16 images and corresponding labels;
Dividing the data set: all image paths and labels are arranged randomly, and the data set is divided into a training set and a testing set.
In a second aspect, the invention provides a road isolation belt detection method based on an image global receptive field, which comprises the following steps:
Acquiring a road isolation belt image to be detected;
Processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to the model training method of the first aspect.
In a third aspect, the present invention provides a road isolation belt detection device based on an image global receptive field, including:
The acquisition module is used for acquiring the road isolation belt image to be detected;
the processing module is used for processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to the model training method of the first aspect.
In a fourth aspect, the present invention provides a non-transitory computer readable storage medium for storing computer instructions which, when executed by a processor, implement the road median strip detection method based on image global receptive field according to the second aspect.
In a fifth aspect, the present invention provides a computer device, comprising a memory and a processor, the processor and the memory being in communication with each other, the memory storing program instructions executable by the processor, the processor invoking the program instructions to perform the image global receptive field-based road barrier detection method according to the second aspect.
In a sixth aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the instructions for implementing the road median strip detection method based on the image global receptive field according to the second aspect.
Term interpretation:
Receptive field (RECEPTIVE FIELD): the receptive field refers to the sensing range of a certain layer of neurons in the neural network to input data. In Convolutional Neural Networks (CNNs), receptive fields represent the size of the input image region of interest to a neuron. Smaller receptive fields are typically used to capture local features, while larger receptive fields are used to understand global structure and context information.
Residual neural network (Residual Neural Network, resNet): the residual neural network is a deep learning architecture and aims to solve the problems of gradient elimination and gradient explosion in the deep neural network. It allows jump connections by introducing a Residual Block (Residual Block) structure, making the network deeper and easier to train. The key innovation of ResNet is that identity mapping is used as a reference model, and further layers can learn residual information through residual connection, so that network performance is improved.
Road isolation belt: road isolation belts are a type of traffic facility on roads, typically areas for separating different directions or lanes of travel, and may take different forms, such as solid lines, broken lines, fences, cement or green belts, etc. Road barriers help to improve traffic safety, guide vehicle flow, and separate different types of traffic.
Slow traffic: slow traffic refers to a slower mode of traffic in cities or roads, typically including walking, riding bicycles, skateboards, and other non-motorized vehicles, as well as pedestrians. Slow traffic plays an important role in urban sustainable development and reducing traffic congestion.
The invention has the beneficial effects that: different from the prior network model, the residual neural network technology based on the global receptive field can represent the characteristics on a level with finer granularity through the proposed multi-scale characteristic enhancement module, the receptive field of a network layer is increased, and a more accurate end-to-end machine non-isolation belt type identification technology and device can be developed. By inputting a traffic image, reliable prediction output can be obtained, and the form of the mechanical non-isolation zone of the scene can be accurately judged, including but not limited to a fence, a solid line, a green belt or other forms, so that the comprehensiveness and the accuracy of the recognition result are improved, and an effective and innovative solution is provided for solving the problem of mechanical non-isolation zone recognition under a complex road scene.
The advantages of additional aspects of the invention will be set forth in part in the description which follows, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a road isolation belt detection model training method based on an image global receptive field according to an embodiment of the invention.
Fig. 2 is a flowchart of a road isolation belt detection method based on an image global receptive field according to an embodiment of the invention.
Fig. 3 is a schematic diagram of a residual network structure of ResNet in the prior art.
Fig. 4 is a schematic diagram of an improved ResNet residual network structure according to an embodiment of the present invention.
FIG. 5 is an image of a labeled inspection result output by a model according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements throughout or elements having like or similar functionality. The embodiments described below by way of the drawings are exemplary only and should not be construed as limiting the invention.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, and/or groups thereof.
In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.
In order that the invention may be readily understood, a further description of the invention will be rendered by reference to specific embodiments that are illustrated in the appended drawings and are not to be construed as limiting embodiments of the invention.
It will be appreciated by those skilled in the art that the drawings are merely schematic representations of examples and that the elements of the drawings are not necessarily required to practice the invention.
Example 1
In this embodiment 1, there is provided a road isolation belt detection device based on an image global receptive field, including: the acquisition module is used for acquiring the road isolation belt image to be detected; and the processing module is used for processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result.
In this embodiment, based on the above device, a road isolation belt detection method based on an image global receptive field is implemented, including: acquiring a road isolation belt image to be detected by using an acquisition module; processing the acquired image by using a road isolation belt detection model based on the image global receptive field by using a processing module to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to a road isolation belt detection model training method based on the image global receptive field.
In this embodiment, a road isolation belt detection model training method based on an image global receptive field includes: acquiring training data; the training data comprises a plurality of road isolation belt images and labels for labeling isolation belt types in the images; training the road isolation belt detection model based on the image global receptive field based on the training data; the method comprises the steps of obtaining a predicted road isolation belt form through a forward propagation process of layer-by-layer convolution, calculating loss between a predicted result and a real label, and carrying out backward propagation through the loss to update model weights until the set iteration number is reached; the road isolation belt detection model based on the image global receptive field comprises a multi-scale enhancement module, wherein the multi-scale enhancement module is used for reducing the number of input channels by a 1x1 convolution layer and batch normalization, a series of 3x3 convolution layers and batch normalization layers are used for feature extraction and multi-scale information, then the 1x1 convolution layers and batch normalization layers are used for increasing the number of output channels, finally a ReLU activation function is used for reducing the size of an input feature map, and an optional downsampling part is used for reducing the size of the input feature map.
The network structure of the road isolation belt detection model based on the image global receptive field adopts ResNet residual network structure, and a group of smaller filter banks are introduced to replace a single 3X 3 convolution kernel of n channels; wherein each filter bank contains w channels, n=s×w, s being the number of filters in the filter bank; these filter banks are interrelated in a hierarchical fashion like a residual connection, increasing the dimensional diversity of the output features.
For input features, the number of channels is adjusted using a1×1 convolutional layer; dividing the input features into three levels; each group of filters firstly extracts features from one group of input feature graphs, and the output features of the previous group are transmitted to the next group of filters together with the other group of input feature graphs until all the input feature graphs are processed; all sets of feature maps are concatenated together and input into a1 x1 convolutional layer to achieve complete information fusion.
Calculating the loss of each training sample by traversing the training dataset and performing back propagation to update model parameters; at regular intervals of iterations, the model propagates forward through the test dataset, calculating test loss and accuracy to evaluate the performance of the model on unseen data until a specified number of iterations is reached.
Acquiring training data, comprising: the traffic image is divided into different forms of road isolation belts: "./FACILITY STRIP" indicates that the road barrier of the image is cement or green belt, "/fence" indicates that the road barrier of the image is fence, "/line" indicates that the road barrier of the image is solid line and "/else" indicates that the road barrier of the image is in a form other than the above three cases; and reading and processing the data set image by using codes to obtain training data and a test data set.
Reading and processing the dataset image with the code, obtaining training data and test data comprising:
Acquiring file paths of all images in a folder containing image files under a code-specified path by using a glob library, a glob. Glob method;
Four image categories are defined: 'else', 'FACILITY STRIP', 'fence', and 'line', and the mapping relationship between them and integer tags;
Creating a custom data set class MYDATASET which inherits the self-supporting image path, the label and the data transformation function in the initialization process, wherein the custom data set class is used for processing image data;
the data transformation function transform defines a series of transformation operations including resizing the image to 256x256 pixels and transforming the image to a sheet;
creating a data loader dataloader for dividing the dataset into batches, each batch containing 16 images and corresponding labels;
Dividing the data set: all image paths and labels are arranged randomly, and the data set is divided into a training set and a testing set.
Example 2
In this embodiment 2, there is provided a road barrier detection method based on an image global receptive field, which uses a road barrier detection model based on an image global receptive field.
First, in this embodiment, a series of configuration works are required to train the road isolation belt detection model based on the image global receptive field. These configurations include installing the Linux operating system, creating a Python 3.7 or higher version development environment, and configuring the deep learning framework PyTorch 1.4.4 or higher version. Since the algorithms employed are based on deep learning models, it is suggested to perform model training in a GPU environment, requiring installation of PyTorch 1.4.4 or higher versions of GPU versions, and corresponding versions of CUDA parallel computing architecture.
The basic flow of training of the road isolation belt detection model in the slow traffic scene is shown in fig. 1. In the model training stage, a training set constructed by traffic images to be detected is input, and a predicted road isolation belt form is obtained through a forward propagation process of layer-by-layer convolution. Then, the loss between the predicted result and the real label is calculated and back-propagated through the loss to update the model weights. This process loops until a set number of iteration rounds (epoch) is reached.
In the model test phase, test set data is loaded and a trained model is used to generate a prediction result. Subsequently, calculation of evaluation indexes is performed, and model performance is evaluated based on the indexes. If the model fails to meet the expected requirements, the model returns to the training stage for further adjustment and training. Once the expected performance level is reached, the model weights are saved, and the whole technological process is completed, so that the final solution is obtained.
The model obtained by the model training method is used for detecting the road isolation belt form in the traffic image, and as shown in fig. 2, the model adopts a deep learning residual error neural network technology, so that the end-to-end automatic detection of the road isolation belt is realized, and the efficiency of judging the non-isolated form of the road machine in a slow traffic scene is remarkably improved. The innovation enables us to more effectively cope with complex traffic scenes, and provides important support for urban traffic management. A smaller set of filters was introduced instead of a single 3 x 3 convolution kernel of n channels than the original ResNet residual network structure shown in fig. 3. Each filter bank contains w channels, where n=s×w, s being the number of filters in the filter bank, s=3 in this embodiment. These filter banks are correlated in a hierarchical fashion (as shown in fig. 4) like a residual connection to increase the dimensional diversity of the output features, this module being named "Bottleneck-Multi".
Specifically, for the input features, the number of channels is first adjusted using a1×1 convolutional layer. These features are then divided into three levels, as shown in fig. 4. Each set of filters first extracts features from a set of input feature maps. The output features of the previous set are then passed to the next set of filters along with another set of input feature maps. This process is repeated until all the input feature maps have been processed. Finally, all sets of feature maps are concatenated together and input into a1 x 1 convolutional layer to achieve complete information fusion. In the present embodiment, the letter I is used to denote an input feature, and the letter O is used to denote an output feature.
With the module, the features can be directly output through single convolution, or can be output after being added with the input feature images of the next group after single convolution. Thus, the output may be obtained from a single set of input features by a single convolution, or from two sets of input features by different levels of convolution. This allows the resulting output features to be combined more abundantly at different scales, thereby increasing the receptive field of the network layer. In addition, the newly proposed multi-scale feature enhancement module reduces computational complexity and parameter amounts compared to the original residual structure module. In the original network structure, the 3×3 convolution layer brings about relatively high computational complexity. For example, when the number of input channels is 512, the computational complexity of the original network is 512×3×3×512. In contrast, with the new module we propose, the features are divided into three groups, each 3×3 convolution layer only requiring 1/8 of the original computational complexity, thus effectively reducing the computational burden.
Such improvements enhance the receptive field of each network layer while exhibiting multi-scale features at finer granularity levels, providing more accurate performance for identification of roadway isolation zones. This is particularly important for the task of non-isolated band detection, since according to a priori knowledge, isolated bands typically extend through the entire image, and only image-based global receptive fields enable more accurate identification of isolated band forms. The innovative design enables the network model to be more suitable for complex road scenes, improves the accuracy and the robustness of the detection of the non-isolated area of the machine, and brings important breakthrough to the intelligent traffic field.
Specifically, in model training, the inputs to the algorithm are: 1. slow traffic scene image data: the method comprises a training set and a testing set, wherein the training set and the testing set are stored in different folders according to different road isolation belts, and the names of the folders in which images are located are used as real labels and predicted values to calculate loss values; 2. model algorithm hyper-parameters: the method comprises the steps of cutting the images, and the batch size, the iteration round number, the learning rate and the like in training. The output of the algorithm is: the parameter weights of the trained model algorithm reaching the performance evaluation standard are obtained, and the image of the inspection result is marked (shown in fig. 5).
The method comprises the following steps:
And (3) a step of: input image preprocessing stage
Step 1-1: the traffic image is divided into four parts according to different forms of road isolation belts: "FACILITY STRIP" means that the road barrier of such an image is cement or green belt; "/fence" indicates that the road barrier for such images is a fence; "/line" indicates that the road barrier for this type of image is a solid line; "/else" means that the road barrier form of this type of image is other than the above three cases.
Step 1-2: the method for reading and processing the data set image by using codes for the deep learning model mainly comprises the following steps:
(1) Using the glob library, the glob.glob method, file paths of all images in a folder containing image files under a code-specified path are acquired.
(2) Four image categories are defined: 'else', 'FACILITY STRIP', 'fence', and 'line', and the mapping relationship between them and integer tags.
(3) Custom dataset class MYDATASET is created that inherits from the torch. In initialization, the incoming image path, labels, and data transformation (conversion) functions.
(4) The data transformation function transform defines a series of transformation operations including resizing an image to 256x256 pixels and transforming the image to a Tensor (Tensor).
(5) A data loader dataloader is created for dividing the dataset into batches, each batch containing 16 images and corresponding labels. This is to load and process data efficiently.
(6) The data set is partitioned. First, all image paths and labels are randomly arranged (shuffled). The dataset was then divided into training and testing sets, with 80% of the data used for training and 20% used for testing.
(7) Finally, instances of the training data set (train_ds) and the test data set (test_ds) are created, as well as corresponding data loaders (train_dl and test_dl).
2. Model training stage
Step 2-1: a network structure is defined. We first need to define what we newly propose as a Multi-scale enhancement module "Bottleneck-Multi". This is the basic component of the network model for constructing the entire network structure. The module is used for reducing the number of input channels by a 1x1 convolution layer and batch normalization, a series of 3x3 convolution layers and batch normalization layers are used for feature extraction and multi-scale information, then the 1x1 convolution layers and batch normalization layers are used for increasing the number of output channels, finally a ReLU activation function, and an optional downsampling part is used for reducing the size of an input feature map. A new "ResNet" class and model function is then constructed from this module.
Step 2-2: setting an optimizer and a loss function for training a network model and various super parameters. A cross entropy loss function is chosen to define the loss function. An Adam optimizer was used to optimize the weights of the model with a learning rate of 0.003. The loop iterates through a specified number epochs (here 120) to train the model. The model is migrated to the GPU (if available) for accelerated computation.
Step 2-3: the model first calculates the loss of each training sample by traversing the training data set and performs back propagation to update the model parameters, a process that is used to train the model to better fit the training data. The model then propagates forward through the test dataset at regular intervals of iterations, calculating test loss and accuracy to evaluate the performance of the model on unseen data. These steps are alternated until a specified number of iterations is reached (epochs).
Step 2-4: saving the trained model to the file "belt_model.
3. Model prediction stage
Step 3-1: traversing all picture files under a specified folder, and acquiring file names under the folder by using os.
Step 3-2: the picture is loaded and the picture is opened using the PIL library. The image is preprocessed using transforms, adjusted to a tensor of size (256), and normalized. A network model is defined that is structurally identical to the model used in training, but removes the last layer of classifier and modifies the classifier based thereon to match the task. The trained model is loaded, and the model is obtained through training, and the file name is' belt_model.
Step 3-3: and transmitting the processed image to a model for prediction. This procedure ensures that the model is in evaluation mode by model. And predicting through the model to obtain the output probability distribution. And (3) acquiring an index of the prediction result by using the output. Argmax (1), and searching a corresponding category label in the data_class according to the index.
Step 3-4: the prediction is printed out, indicating into which category the image is classified, and the category label is added to the image. Finally, the image with the predicted result is saved in a folder "/result", and the file name is the same as the original image.
Example 3
Embodiment 3 provides a non-transitory computer readable storage medium for storing computer instructions which, when executed by a processor, implement the road median strip detection method based on image global receptive field as described above, the method comprising:
Acquiring a road isolation belt image to be detected by using an acquisition module; processing the acquired image by using a road isolation belt detection model based on the image global receptive field by using a processing module to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to a road isolation belt detection model training method based on the image global receptive field.
Example 4
Embodiment 4 provides a computer device including a memory and a processor, where the processor and the memory are in communication with each other, the memory stores program instructions executable by the processor, and the processor invokes the program instructions to execute the road median strip detection method based on image global receptive field as described above, the method includes:
Acquiring a road isolation belt image to be detected by using an acquisition module; processing the acquired image by using a road isolation belt detection model based on the image global receptive field by using a processing module to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to a road isolation belt detection model training method based on the image global receptive field.
Example 5
Embodiment 5 provides an electronic apparatus including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the instructions for implementing the road median strip detection method based on the image global receptive field as described above, and the method comprises:
Acquiring a road isolation belt image to be detected by using an acquisition module; processing the acquired image by using a road isolation belt detection model based on the image global receptive field by using a processing module to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to a road isolation belt detection model training method based on the image global receptive field.
In summary, the road isolation belt detection model training method and the isolation belt detection method based on the image global receptive field provided by the invention apply the deep learning technology on the detection task of the road isolation belt for the first time, so that the detection process is more automatic, and no large amount of manual identification is relied on, thereby improving the efficiency and the adaptability. The innovation not only enables the road isolation belt to be more convenient to detect, but also is widely applicable to intelligent traffic background, provides a powerful tool for traffic management and intelligent traffic systems, and is expected to obviously improve traffic safety and efficiency. The multi-scale characteristic enhancement module is introduced, so that the representation capability of the model on multi-scale information is improved. Compared with the traditional method, the module can better capture details in the image, so that the detection accuracy of the road isolation belt is improved. The requirement of the road isolation belt recognition task is better met by improving the residual error network structure, particularly in the aspect of receptive fields of models. It is able to handle situations where isolation is typically used throughout the entire image, thus predicting the output more accurately.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While the foregoing description of the embodiments of the present invention has been presented in conjunction with the drawings, it should be understood that it is not intended to limit the scope of the invention, but rather, it should be understood that various changes and modifications could be made by one skilled in the art without the need for inventive faculty, which would fall within the scope of the invention.

Claims (10)

1. The road isolation belt detection model training method based on the image global receptive field is characterized by comprising the following steps of:
Acquiring training data; the training data comprises a plurality of road isolation belt images and labels for labeling isolation belt types in the images;
Training the road isolation belt detection model based on the image global receptive field based on the training data; the method comprises the steps of obtaining a predicted road isolation belt form through a forward propagation process of layer-by-layer convolution, calculating loss between a predicted result and a real label, and carrying out backward propagation through the loss to update model weights until the set iteration number is reached; the road isolation belt detection model based on the image global receptive field comprises a multi-scale enhancement module, wherein the multi-scale enhancement module is used for reducing the number of input channels by a 1x1 convolution layer and batch normalization, a series of 3x3 convolution layers and batch normalization layers are used for feature extraction and multi-scale information, then the 1x1 convolution layers and batch normalization layers are used for increasing the number of output channels, finally a ReLU activation function is used for reducing the size of an input feature map, and an optional downsampling part is used for reducing the size of the input feature map.
2. The image global receptive field-based road isolation belt detection model training method according to claim 1, wherein the network structure of the image global receptive field-based road isolation belt detection model adopts ResNet residual network structure, and a group of smaller filter banks is introduced to replace a single 3 x 3 convolution kernel of n channels; wherein each filter bank contains w channels, n=s×w, s being the number of filters in the filter bank; these filter banks are interrelated in a hierarchical fashion like a residual connection, increasing the dimensional diversity of the output features.
3. The image global receptive field-based road isolation belt detection model training method of claim 2, wherein for input features, the number of channels is adjusted by using a1 x 1 convolution layer; dividing the input features into three levels; each group of filters firstly extracts features from one group of input feature graphs, and the output features of the previous group are transmitted to the next group of filters together with the other group of input feature graphs until all the input feature graphs are processed; all sets of feature maps are concatenated together and input into a1 x 1 convolutional layer to achieve complete information fusion.
4. The image global receptive field based road isolation belt detection model training method of claim 1, wherein model parameters are updated by traversing a training dataset, calculating a penalty for each training sample and performing back propagation; at regular intervals of iterations, the model propagates forward through the test dataset, calculating test loss and accuracy to evaluate the performance of the model on unseen data until a specified number of iterations is reached.
5. The image global receptive field-based road isolation belt detection model training method of claim 1, wherein obtaining training data comprises: the traffic image is divided into different forms of road isolation belts: "./FACILITY STRIP" indicates that the road barrier of the image is cement or green belt, "/fence" indicates that the road barrier of the image is fence, "/line" indicates that the road barrier of the image is solid line and "/else" indicates that the road barrier of the image is in a form other than the above three cases; and reading and processing the data set image by using codes to obtain training data and a test data set.
6. The method of training a road isolation belt detection model based on an image global receptive field according to claim 5, wherein reading and processing the dataset image with codes to obtain training data and test data comprises:
Acquiring file paths of all images in a folder containing image files under a code-specified path by using a glob library, a glob. Glob method;
Four image categories are defined: 'else', 'FACILITY STRIP', 'fence', and 'line', and the mapping relationship between them and integer tags;
Creating a custom data set class MYDATASET which inherits the self-supporting image path, the label and the data transformation function in the initialization process, wherein the custom data set class is used for processing image data;
the data transformation function transform defines a series of transformation operations including resizing the image to 256x256 pixels and transforming the image to a sheet;
creating a data loader dataloader for dividing the dataset into batches, each batch containing 16 images and corresponding labels;
Dividing the data set: all image paths and labels are arranged randomly, and the data set is divided into a training set and a testing set.
7. The road isolation belt detection method based on the image global receptive field is characterized by comprising the following steps of:
Acquiring a road isolation belt image to be detected;
Processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to the model training method of any one of claims 1-6.
8. Road median detection device based on image global receptive field, characterized by comprising:
The acquisition module is used for acquiring the road isolation belt image to be detected;
the processing module is used for processing the acquired image by using a road isolation belt detection model based on the image global receptive field to obtain a road isolation belt type detection result; the road isolation belt detection model based on the image global receptive field is obtained according to the model training method of any one of claims 1-6.
9. A computer device comprising a memory and a processor, the processor and the memory in communication with each other, the memory storing program instructions executable by the processor, the processor invoking the program instructions to perform the image global receptive field based road barrier detection method of claim 7.
10. An electronic device, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the instructions for implementing the road median strip detection method based on the image global receptive field as defined in claim 7.
CN202410105601.2A 2024-01-25 2024-01-25 Road isolation belt detection method and device based on image global receptive field Pending CN118155161A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410105601.2A CN118155161A (en) 2024-01-25 2024-01-25 Road isolation belt detection method and device based on image global receptive field

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410105601.2A CN118155161A (en) 2024-01-25 2024-01-25 Road isolation belt detection method and device based on image global receptive field

Publications (1)

Publication Number Publication Date
CN118155161A true CN118155161A (en) 2024-06-07

Family

ID=91287562

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410105601.2A Pending CN118155161A (en) 2024-01-25 2024-01-25 Road isolation belt detection method and device based on image global receptive field

Country Status (1)

Country Link
CN (1) CN118155161A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110298266A (en) * 2019-06-10 2019-10-01 天津大学 Deep neural network object detection method based on multiple dimensioned receptive field Fusion Features
US20200134772A1 (en) * 2018-10-31 2020-04-30 Kabushiki Kaisha Toshiba Computer vision system and method
CN111797803A (en) * 2020-07-15 2020-10-20 郑州昂达信息科技有限公司 Road guardrail abnormity detection method based on artificial intelligence and image processing
US20200410273A1 (en) * 2018-08-24 2020-12-31 Tencent Technology (Shenzhen) Company Limited Target detection method and apparatus, computer-readable storage medium, and computer device
CN114419180A (en) * 2022-01-25 2022-04-29 智道网联科技(北京)有限公司 Method and device for reconstructing high-precision map and electronic equipment
CN114612832A (en) * 2022-03-14 2022-06-10 百果园技术(新加坡)有限公司 Real-time gesture detection method and device

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20200410273A1 (en) * 2018-08-24 2020-12-31 Tencent Technology (Shenzhen) Company Limited Target detection method and apparatus, computer-readable storage medium, and computer device
US20200134772A1 (en) * 2018-10-31 2020-04-30 Kabushiki Kaisha Toshiba Computer vision system and method
CN110298266A (en) * 2019-06-10 2019-10-01 天津大学 Deep neural network object detection method based on multiple dimensioned receptive field Fusion Features
CN111797803A (en) * 2020-07-15 2020-10-20 郑州昂达信息科技有限公司 Road guardrail abnormity detection method based on artificial intelligence and image processing
CN114419180A (en) * 2022-01-25 2022-04-29 智道网联科技(北京)有限公司 Method and device for reconstructing high-precision map and electronic equipment
CN114612832A (en) * 2022-03-14 2022-06-10 百果园技术(新加坡)有限公司 Real-time gesture detection method and device

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
张邯,罗晓曙,袁荣尚: "基于优化的卷积神经网络在交通标志识别中的应用", 《现代电子技术》, 1 November 2018 (2018-11-01), pages 132 - 136 *
王益琴: "《基于深度学习的遥感图像语义分割方法研究》", 31 December 2022, 科学技术文献出版社, pages: 12 - 13 *

Similar Documents

Publication Publication Date Title
Zhou et al. Split depth-wise separable graph-convolution network for road extraction in complex environments from high-resolution remote-sensing images
CN112200161B (en) Face recognition detection method based on mixed attention mechanism
Chen et al. DenseLightNet: A light-weight vehicle detection network for autonomous driving
Huang et al. Spatial-temproal based lane detection using deep learning
Rahman et al. Densely-populated traffic detection using YOLOv5 and non-maximum suppression ensembling
CN111104903A (en) Depth perception traffic scene multi-target detection method and system
CN114359130A (en) Road crack detection method based on unmanned aerial vehicle image
CN112861931B (en) Multi-level change detection method, system, medium and electronic device based on difference attention neural network
Zhao et al. Multiple vision architectures-based hybrid network for hyperspectral image classification
CN113034506A (en) Remote sensing image semantic segmentation method and device, computer equipment and storage medium
Luo et al. AD-RoadNet: an auxiliary-decoding road extraction network improving connectivity while preserving multiscale road details
Wu et al. Research on asphalt pavement disease detection based on improved YOLOv5s
Pramanik et al. Detection of Potholes using Convolutional Neural Network Models: A Transfer Learning Approach
Dixit et al. Automatic building extraction from high-resolution satellite images using deep learning techniques
CN116977747B (en) Small sample hyperspectral classification method based on multipath multi-scale feature twin network
AU2019100967A4 (en) An environment perception system for unmanned driving vehicles based on deep learning
CN118155161A (en) Road isolation belt detection method and device based on image global receptive field
Lan et al. Crack detection based on attention mechanism with YOLOv5
CN116935249A (en) Small target detection method for three-dimensional feature enhancement under unmanned airport scene
Thayammal et al. Analysis of water body segmentation from Landsat imagery using deep neural network
Li et al. A YOLOv4 model with FPN for service plates detection
Li et al. Prediction model of urban street public space art design indicators based on deep convolutional neural network
WO2022197958A1 (en) Frames alignment in video processing using self-attention
Phi et al. Evaluation Of Vision Transformer On Weather Image Recognition
Nataprawira et al. Pedestrian Detection in Different Lighting Conditions Using Deep Neural Networks.

Legal Events

Date Code Title Description
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination