CN115937314A - Camellia oleifera fruit growth posture detection method - Google Patents
Camellia oleifera fruit growth posture detection method Download PDFInfo
- Publication number
- CN115937314A CN115937314A CN202211661768.4A CN202211661768A CN115937314A CN 115937314 A CN115937314 A CN 115937314A CN 202211661768 A CN202211661768 A CN 202211661768A CN 115937314 A CN115937314 A CN 115937314A
- Authority
- CN
- China
- Prior art keywords
- fruit
- oil tea
- fruits
- posture
- growth
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method for detecting the growth posture of oil tea fruits, which comprises the following steps: acquiring an RGB image and a three-dimensional point cloud of the oil tea fruit by using a binocular camera; marking out a circumscribed rectangular frame and a corresponding growth posture category of each fruit in the image, and constructing an image data set; training a camellia oleifera fruit position and posture recognition model based on a deep learning target detection method by adopting the image data set; calculating an external rectangular frame of the oil tea fruit in the image and a rough growth posture expression of the external rectangular frame by using the trained model; accurately extracting a fruit area in an external rectangular frame by adopting an ellipse fitting algorithm and an adaptive threshold segmentation algorithm, and acquiring a point set P corresponding to a three-dimensional subdata point set representing fruits; and calculating a refined fruit growth posture expression according to the rough fruit growth posture expression and the characteristic points in the point set P. The method has the advantages of small data processing amount, relatively simple steps and high detection efficiency, and can provide technical support for realizing automatic oil tea fruit picking with low bud damage rate.
Description
Technical Field
The invention relates to the field of machine vision and artificial intelligence, in particular to a method for detecting the growth posture of oil tea fruits.
Background
The oil tea is a woody oil tree species, is one of four oil source trees in the world, and the fruit of the oil tea is regarded as a healthy edible oil raw material and is planted in hilly areas in south China. The problems of disordered distribution of branches of the camellia oleifera, synchronous flowers and fruits, intensive growth of fruits, irregular direction and the like make the efficient harvesting of the camellia oleifera difficult.
Traditional oil tea fruit is gathered and mainly relies on manual operation, wastes time and energy, with high costs. With the development of robotics and advanced agriculture and forestry machinery, the oil tea fruit harvesting robot is widely applied to oil tea fruit harvesting. At the present stage, equipment such as a shaking oil tea fruit picking robot, a combing oil tea fruit picking robot and the like can be applied, and a good picking effect is obtained; however, due to the need of separating fruits and fruit stalks, brushing in a large range or mechanical vibration of the whole camellia oleifera tree can damage a large number of buds, flowers, leaves and other parts of branches and fruits at the same time, and the yield of the camellia oleifera fruits can be obviously reduced; the shaking type camellia oleifera fruit picking robot and the combing type camellia oleifera fruit picking robot are limited in practical application due to the problems.
With the development of machine vision technology, a picking method based on camellia oleifera fruit visual positioning appears. The method comprises the steps of shooting an image of the oil tea fruit by using a common camera or a depth camera, calculating the position of the oil tea fruit by using an intelligent image processing method, and guiding a mechanical arm to accurately grab the fruit. Although the method can realize accurate picking, the problems that the growth direction of the oil tea fruits is irregular, leaves and flowers shield the fruits and the like cause that surrounding branches and buds can be seriously damaged when the fruits are picked by using modes of twisting, shearing and the like.
The development of artificial intelligence technology brings a new method for fruit harvesting. The combination of deep learning and machine vision can realize the identification of the spatial positions of fruits and fruit stalks. The method is primarily applied to picking of fruits such as apples and grapes, wherein the stems of the fruits are long and obvious, and the fruits grow towards the ground along with the gravity. However, the fruit stalks of the oil-tea camellia fruits are short, the growth angle is uncertain, and the intelligent fruit stalk identification method is difficult to directly apply.
Disclosure of Invention
The invention aims to provide a method for detecting the growth posture of oil tea fruits, which can solve the problems of complex data processing steps, low efficiency and high labor labeling cost of the existing intelligent fruit posture detection method, and can provide a technical basis for realizing automatic oil tea fruit picking with low bud damage rate.
In order to achieve the purpose, the method for detecting the growth posture of the camellia oleifera fruits comprises the following steps:
(1) Acquiring RGB images and three-dimensional point cloud data of the oil tea fruits by using a binocular camera;
(2) Carrying out manual labeling on the RGB image of the oil tea fruit, and labeling out an external rectangular frame and corresponding categories of each fruit in the image: five growth posture categories q, namely, upper, lower, left, right and front, are not marked with fruits growing backwards, and an image data set is constructed;
(3) And training a deep learning target detection model FasterRCNN by adopting the image data set, and obtaining a camellia oleifera fruit position and posture recognition model after training.
(4) Directly processing the RGB image of the oil tea fruit by using the oil tea fruit position and posture recognition model, and outputting pixel coordinates of an external rectangular frame of the oil tea fruit in the RGB image and a rough growth posture expression of the oil tea fruit;
(5) Extracting a rectangular area of the detected fruit target, acquiring sub-images in the corresponding area of the RGB image (acquiring the sub-images in the rectangular area of the fruit target in the RGB image), and acquiring sub-data point sets in the corresponding area in the three-dimensional point cloud data;
(6) And accurately extracting pixel points covered by fruits in the subimages by adopting an ellipse fitting algorithm and an adaptive threshold segmentation algorithm, and acquiring a point set P representing the fruits in the corresponding subdata point set.
(7) Expressing the actual position of the fruit by using data point space coordinates (X, Y, Z) in a point cloud subset corresponding to the pixel coordinates of the center point of a circumscribed rectangular area of the fruit target, calculating refined fruit growth posture expressions (E, S) according to the formula F (P, q), and combining the expressions into (X, Y, Z, E, S):
(8) And (5) repeating the steps (5) to (7) to obtain the position and posture expressions of all fruits in the image.
(9) According to the spatial position and the growth posture expression (X, Y, Z, E and S) of the fruit, the positions and the directions of the mechanical arms and the tail end mechanical paw are adjusted, so that the paw can move and grab along the central axis direction of the camellia oleifera fruit, and the damage to branches and buds is reduced.
Preferably, in step (2), the fruit growing rearward is not marked, and the specific judgment basis is: fruits which have an included angle of more than 20 degrees with the imaging plane of the camera and the growing direction pointing to the direction far away from the lens of the camera are not picked; the fruit in other growth states is divided into five growth posture categories of 'upper', 'lower', 'left', 'right' and 'front', and the five growth posture categories are shown in figure 3. In the step (3), the output of the FasterRCNN model is configured as the coordinates of the upper left corner and the lower right corner of the circumscribed rectangular frame of each oil tea fruit in the image, and the corresponding growth posture category, which is specifically shown in fig. 5.
The invention has the beneficial effects that: the method for detecting the growth posture of the oil tea fruits has the advantages of small data processing amount, relatively simple steps and high detection efficiency, and can provide technical support for automatic oil tea fruit picking with low bud damage rate.
Drawings
FIG. 1 is a system framework diagram of a detection method for growth postures of oil tea fruits in the embodiment of the invention;
FIG. 2 is a flow chart of the method for detecting the growth attitude of Camellia oleifera fruit in the embodiment of the present invention;
FIG. 3 is a schematic diagram of a fruit rough growth posture category labeling method used in the embodiment of the present invention; wherein a is a schematic diagram of 5 growth posture categories q, and b is a schematic diagram of artificially labeling growth posture categories for a specific RGB image;
FIG. 4 is a schematic diagram of a fruit refining growth gesture calculation method employed in an embodiment of the present invention; wherein a is the oil tea fruit with the direction indicated by (E, S) being the front direction, and b is the oil tea fruit with the direction indicated by (E, S) being the right direction;
fig. 5 is a structure diagram of a recognition model of the position and posture of the camellia oleifera fruit adopted in the embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described with reference to the following embodiments and accompanying drawings.
Examples
Referring to fig. 1 and 2, the method for detecting the growth posture of the camellia oleifera fruit in the embodiment includes the following steps:
the first step is as follows: and collecting the oil tea fruit image and the three-dimensional point cloud to construct an oil tea fruit data set. A binocular camera 5 is used to acquire a large number of RGB images of the oil-tea camellia fruits and three-dimensional point cloud data. The RGB image is manually marked, the circumscribed rectangle of each fruit in the RGB image and the corresponding rough growth posture class q are marked, and the specific judgment rule is shown in figure 3, upper: fruits which are upward along the gravity direction and have an included angle of less than 20 degrees with the gravity direction; "lower": fruits which are downward along the gravity direction and have an included angle of less than 20 degrees with the gravity direction; "left": fruits with an included angle of less than 20 degrees are arranged leftwards on the imaging plane of the camera and in the direction vertical to the gravity direction; "right": fruits with an included angle of less than 20 degrees are rightwards arranged on the imaging plane of the camera and in the direction vertical to the gravity direction; "front": fruit perpendicular to the camera imaging plane, pointing along the fruit towards the imaging device, with an angle <20 °.
The second step is that: and establishing a deep learning target detection network model. The constructed oil tea fruit data set is stored in a computer terminal, and then the FasterRCNN network is trained by adopting the training data set, and the structure of the FasterRCNN network is shown in figure 5. A ResNet-50 backbone network in FasterRCNN is used for automatically learning and extracting depth features in an image, a region recommendation network and an ROI pooling module acquire features related to fruit information in the image from the extracted depth features, and then a mapping relation is established between the fruit related features and expected output by adopting a multi-layer sensing machine head and two full-connection layers, and the mapping relation is used for predicting fruit posture categories and position coordinates respectively. Based on the process, an end-to-end mapping relation is established, wherein the RGB image is used as input, the camellia oleifera fruit external rectangular frame and the corresponding rough growth posture class q are used as output.
The third step: and detecting the spatial position and the rough growth posture category of the oil tea fruits. After the model is trained at the PC end, the structure and weight parameters of the model are transplanted and solidified on the controller. A set of RGB images 4 of the oil tea fruit and corresponding three-dimensional point cloud data are acquired using a binocular camera 5. And analyzing and processing the RGB image 4 by using the trained model to obtain an external rectangular frame 6 of the camellia oleifera fruit 3 and a corresponding rough growth posture class q.
The fourth step: and calculating the growth posture information of the refined oil tea fruits. In the circumscribed rectangle frame 6, an ellipse fitting algorithm and a self-adaptive threshold segmentation algorithm are adopted to accurately extract a pixel area 7 covered by fruits in the subimages, and a three-dimensional point cloud subset P representing the fruits in a corresponding subdata point set is obtained. Calculating a refined fruit growth posture expression 8 according to the formula F (P, q), defined as (E, S), indicating that the point E points to the direction of the point S:
the fifth step: and determining the picking direction of the mechanical gripper according to the spatial position and the growth posture information of the fruit. And (4) searching a three-dimensional space coordinate corresponding to the pixel coordinate of the central point of the circumscribed rectangular frame 6 in the three-dimensional point cloud subset P, and defining the three-dimensional space coordinate as (X, Y, Z), thus obtaining the actual space position of the oil-tea camellia fruit. And fusing the information of the actual spatial position (X, Y, Z) of the oil-tea camellia fruit and the fruit growth posture expression (E, S) obtained by calculation, and controlling the movement of the mechanical gripper 2. And (E) taking the vector direction indicated by the (E, S) as the target orientation of the central axis 1 of the mechanical paw, controlling the mechanical paw 2 to reach the specified position, and picking the oil-tea camellia fruits according to the specified angle.
The invention discloses a method for detecting the growth posture of oil tea fruits, which belongs to the field of machine vision and artificial intelligence and comprises the following steps: acquiring an RGB image and a three-dimensional point cloud of the oil tea fruit by using a binocular camera; marking out a circumscribed rectangular frame and a corresponding growth posture category of each fruit in the image, and constructing an image data set; training a camellia oleifera fruit position and posture recognition model based on a deep learning target detection method by adopting the image data set; calculating a circumscribed rectangular frame of the oil tea fruits in the image and a rough growth posture expression of the circumscribed rectangular frame by using the trained model; accurately extracting a fruit region in an external rectangular frame by adopting an ellipse fitting algorithm and a self-adaptive threshold segmentation algorithm, and acquiring a point set P which corresponds to a three-dimensional subdata point set and represents a fruit; calculating a refined fruit growth posture expression according to the rough fruit growth posture expression and the characteristic points in the point set P; and controlling the mechanical gripper to move to a target fruit position, move along the central axis direction of the target fruit and grab according to the spatial position and the growth posture expression of the fruit. The method can solve the problems of complex data processing steps, low efficiency and high labor labeling cost of the existing intelligent fruit posture detection method, and can provide a technical basis for realizing automatic oil tea fruit picking with low bud damage rate.
Claims (4)
1. A method for detecting the growth posture of oil tea fruits is characterized by comprising the following steps: the method comprises the following steps:
(1) Acquiring RGB images and three-dimensional point cloud data of the oil tea fruits by using a binocular camera;
(2) Carrying out manual labeling on the RGB image of the oil tea fruit, and labeling out an external rectangular frame and corresponding categories of each fruit in the image: five growth posture categories q, namely, upper, lower, left, right and front, are not marked with fruits growing backwards, and an image data set is constructed;
(3) Training a deep learning target detection model FasterRCNN by using the image data set, and obtaining a camellia oleifera fruit position and posture recognition model after training;
(4) Directly processing the RGB image of the oil tea fruit by using the oil tea fruit position and posture recognition model, and outputting pixel coordinates of an external rectangular frame of the oil tea fruit in the RGB image and rough growth posture expression of the oil tea fruit;
(5) Extracting a rectangular area of the detected fruit target, acquiring sub-images in the corresponding area of the RGB image, and acquiring a sub-data point set in the corresponding area in the three-dimensional point cloud data;
(6) Accurately extracting pixel points covered by fruits in the subimages by adopting an ellipse fitting algorithm and a self-adaptive threshold segmentation algorithm, and acquiring a point set P representing the fruits in the corresponding subdata point set;
(7) And (3) expressing the actual position of the fruit by using the data point space coordinates (X, Y, Z) in a point set P corresponding to the pixel coordinates of the center point of the circumscribed rectangular region of the fruit target, calculating a refined fruit growth posture expression (E, S) according to the formula F (P, q), and combining the expression (X, Y, Z, E, S):
(8) And (5) repeating the steps (5) to (7) to obtain the position and posture expressions of all fruits in the image.
2. The camellia oleifera fruit growth posture detection method as claimed in claim 1, wherein: in the step (2), fruits growing backwards are not marked, and the concrete judgment basis is as follows: fruit which has an angle of more than 20 degrees with the imaging plane of the camera and the growing direction points to the direction far away from the lens of the camera.
3. The camellia oleifera fruit growth posture detection method as claimed in claim 1, wherein: in the step (3), the output of the FasterRCNN model is configured into the coordinates of the upper left corner and the lower right corner of a circumscribed rectangular frame of each oil tea fruit in the image and the corresponding growth posture category.
4. The camellia oleifera fruit growth posture detection method as claimed in claim 1, wherein: the step (8) is followed by the steps of:
(9) According to the spatial position and the growth posture expression (X, Y, Z, E, S) of the fruit, the position and the direction of a mechanical arm and a tail end mechanical claw for picking the oil-tea camellia fruit are adjusted, the (X, Y, Z) information is the target position of the central point of the mechanical claw, and the vector direction indicated by the (E, S) information is used as the target orientation of the axis of the mechanical claw, so that the mechanical claw moves along the central axis direction of the oil-tea camellia fruit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211661768.4A CN115937314B (en) | 2022-12-23 | 2022-12-23 | Method for detecting growth posture of oil tea fruits |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211661768.4A CN115937314B (en) | 2022-12-23 | 2022-12-23 | Method for detecting growth posture of oil tea fruits |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115937314A true CN115937314A (en) | 2023-04-07 |
| CN115937314B CN115937314B (en) | 2023-09-08 |
Family
ID=86648882
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211661768.4A Active CN115937314B (en) | 2022-12-23 | 2022-12-23 | Method for detecting growth posture of oil tea fruits |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115937314B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117011843A (en) * | 2023-08-09 | 2023-11-07 | 仲恺农业工程学院 | Image recognition and posture assessment method for automatic picking of dragon fruits |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109978949A (en) * | 2019-03-26 | 2019-07-05 | 南开大学 | A kind of method that crops identification based on computer vision is extracted with characteristic point three-dimensional coordinate |
| CN111445523A (en) * | 2020-03-25 | 2020-07-24 | 中国农业科学院农业信息研究所 | Fruit pose calculation method and device, computer equipment and storage medium |
| US20220005209A1 (en) * | 2018-11-27 | 2022-01-06 | Nanjing Forestry University | Binocular-vision-based method for tracking fruit space attitude and fruit space motion |
| CN114029945A (en) * | 2021-10-13 | 2022-02-11 | 大连理工大学 | Grabbing path control method of spherical-like fruit picking mechanical arm |
| CN114332689A (en) * | 2021-12-14 | 2022-04-12 | 中国科学院合肥物质科学研究院 | Citrus identification and positioning method, device, equipment and storage medium |
| CN114782360A (en) * | 2022-04-19 | 2022-07-22 | 浙江工业大学 | Real-time tomato posture detection method based on DCT-YOLOv5 model |
| CN115250745A (en) * | 2022-08-05 | 2022-11-01 | 华南农业大学 | Full-automatic fruit picking robot and picking method based on vision technology |
| CN115496979A (en) * | 2022-09-15 | 2022-12-20 | 常州大学 | Orchard young fruit growth posture visual identification method based on multiple feature fusion |
| CN115497089A (en) * | 2022-09-30 | 2022-12-20 | 五邑大学 | Dragon fruit picking method, device, equipment and storage medium based on deep learning |
-
2022
- 2022-12-23 CN CN202211661768.4A patent/CN115937314B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220005209A1 (en) * | 2018-11-27 | 2022-01-06 | Nanjing Forestry University | Binocular-vision-based method for tracking fruit space attitude and fruit space motion |
| CN109978949A (en) * | 2019-03-26 | 2019-07-05 | 南开大学 | A kind of method that crops identification based on computer vision is extracted with characteristic point three-dimensional coordinate |
| CN111445523A (en) * | 2020-03-25 | 2020-07-24 | 中国农业科学院农业信息研究所 | Fruit pose calculation method and device, computer equipment and storage medium |
| CN114029945A (en) * | 2021-10-13 | 2022-02-11 | 大连理工大学 | Grabbing path control method of spherical-like fruit picking mechanical arm |
| CN114332689A (en) * | 2021-12-14 | 2022-04-12 | 中国科学院合肥物质科学研究院 | Citrus identification and positioning method, device, equipment and storage medium |
| CN114782360A (en) * | 2022-04-19 | 2022-07-22 | 浙江工业大学 | Real-time tomato posture detection method based on DCT-YOLOv5 model |
| CN115250745A (en) * | 2022-08-05 | 2022-11-01 | 华南农业大学 | Full-automatic fruit picking robot and picking method based on vision technology |
| CN115496979A (en) * | 2022-09-15 | 2022-12-20 | 常州大学 | Orchard young fruit growth posture visual identification method based on multiple feature fusion |
| CN115497089A (en) * | 2022-09-30 | 2022-12-20 | 五邑大学 | Dragon fruit picking method, device, equipment and storage medium based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| DANQI XU等: "3D Reconstruction of camellia Oleifera Fruit Recongnition and Fruit Branch on Kinect Camera", 2021 2ND INTERNATIONAL ON ARTIFICIAL INTELLIGENCE AND INFORMATION SYSTEMS * |
| 李立君 等: "基于改进凸壳理论的遮挡油茶果定位检测算法", 农业机械学报, vol. 47, no. 12 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117011843A (en) * | 2023-08-09 | 2023-11-07 | 仲恺农业工程学院 | Image recognition and posture assessment method for automatic picking of dragon fruits |
| CN117011843B (en) * | 2023-08-09 | 2024-05-14 | 仲恺农业工程学院 | Image recognition and posture assessment method for automatic picking of dragon fruits |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115937314B (en) | 2023-09-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Detection of fruit-bearing branches and localization of litchi clusters for vision-based harvesting robots | |
| Chen et al. | A YOLOv3-based computer vision system for identification of tea buds and the picking point | |
| Zhaoxin et al. | Design a robot system for tomato picking based on YOLO v5 | |
| Wang et al. | Deep learning approach for apple edge detection to remotely monitor apple growth in orchards | |
| CN111602517A (en) | Distributed visual active perception method for string-type fruits and application of distributed visual active perception method | |
| CN201600330U (en) | System for recognizing and locating mature pineapples | |
| Miao et al. | Efficient tomato harvesting robot based on image processing and deep learning | |
| CN111418349B (en) | Intelligent fruit picking robot and method for realizing fruit picking | |
| Kalampokas et al. | Grape stem detection using regression convolutional neural networks | |
| Jin et al. | Detection method for table grape ears and stems based on a far-close-range combined vision system and hand-eye-coordinated picking test | |
| CN108064560A (en) | The automatic picker system of fruit and method based on Kinect depth of field cameras | |
| CN112990103A (en) | String mining secondary positioning method based on machine vision | |
| Badhan et al. | Real-time weed detection using machine learning and stereo-vision | |
| CN114260895B (en) | Method and system for determining grabbing obstacle avoidance direction of mechanical arm of picking robot | |
| Ji et al. | Research on key technology of truss tomato harvesting robot in greenhouse | |
| CN113920190B (en) | Ginkgo flower spike orientation method and system | |
| CN114387343A (en) | Method for detecting picking position of mushroom stem of clustered oyster mushroom | |
| CN115316129A (en) | Self-adaptive bionic picking device based on binocular vision recognition and cluster fruit picking method | |
| Yang et al. | Vision based fruit recognition and positioning technology for harvesting robots | |
| Wang et al. | A research for intelligent cotton picking robot based on machine vision | |
| Rao | Design of automatic cotton picking robot with Machine vision using Image Processing algorithms | |
| CN115937314B (en) | Method for detecting growth posture of oil tea fruits | |
| CN117197595A (en) | Fruit tree growth period identification method, device and management platform based on edge calculation | |
| He et al. | Research Status and Development Trend of Key Technologies for Pineapple Harvesting Equipment: A Review | |
| CN112329697B (en) | Improved YOLOv 3-based on-tree fruit identification method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |