[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN106228162B - A kind of quick object identification method of mobile robot based on deep learning - Google Patents

A kind of quick object identification method of mobile robot based on deep learning Download PDF

Info

Publication number
CN106228162B
CN106228162B CN201610581928.2A CN201610581928A CN106228162B CN 106228162 B CN106228162 B CN 106228162B CN 201610581928 A CN201610581928 A CN 201610581928A CN 106228162 B CN106228162 B CN 106228162B
Authority
CN
China
Prior art keywords
picture
unit
data
depth
object identification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610581928.2A
Other languages
Chinese (zh)
Other versions
CN106228162A (en
Inventor
王威
谈笑
胡义轩
袁泽寰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Fudian Culture Communication Co.,Ltd.
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Priority to CN201610581928.2A priority Critical patent/CN106228162B/en
Publication of CN106228162A publication Critical patent/CN106228162A/en
Application granted granted Critical
Publication of CN106228162B publication Critical patent/CN106228162B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00Arrangements for image or video recognition or understanding
    • G06V10/40Extraction of image or video features
    • G06V10/44Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a kind of quick object identification methods of the mobile robot based on deep learning, and this method comprises the following steps: 1) mobile picture acquisition, 2) image data pretreatment, 3) picture feature extraction, 4) picture prediction output, 5) environmental constraints optimization, 6) picture recognition output.The present invention overcomes traditional object identification system to need to add using object identification the complexity and unstability of detection by the unified integration of detection and recognition result;By the generation of multilayer residual error network design and surrounding gravity constraint condition, the shortcomings that overcoming integrated form object identification system poor accuracy;The integration of detection identification mission also ensures the treatment effeciency of system, improves the sensing capability in robot moving process.

Description

A kind of quick object identification method of mobile robot based on deep learning
Technical field
The present invention relates to a kind of quick object identification method of mobile robot more particularly to a kind of shiftings based on deep learning The quick object identification method of mobile robot.
Background technique
Currently there are object detection identifying systems generally to recommend skill by sliding window or object area first Art obtain an object set of candidate regions, then using didactic Feature Selection come to this gather in candidate region into Row identification, identification refer to by classifier to candidate region it is specified most close to object category.
The difference of feature is used according to classifier, detection system can probably be divided into two types at present: being based on deep learning Mode;Mode based on Heuristic Feature.Scheme based on Heuristic Feature is generally by the artificial design feature of experience come table Show candidate region, and the mode based on deep learning, object features are extracted by multilayer neural network, to reach pair The multiple combination of object local feature, to achieve the effect that classify to object.Deep learning feature can be according to data Distribution carries out adaptive learning, so depth learning technology had on Detection accuracy compared to Heuristic Feature technology in recent years Some superiority.However discovery learning technology is because of its quick and easy inherent characteristic, the still quilt in some simple applications It uses, such as Face datection.
However this two technologies still remain disadvantage at present, can not be applied in actual scene, major embodiment are as follows: heuristic Although quickly, feature representation ability is limited for characterization method.In reality scene, since identified object is typically found in whole picture Among scene, and article size size changes greatly at random, blocks simultaneously because multiple objects exist between each other, so object Body Detection accuracy is relatively low;Although still further aspect depth learning technology can be very good to indicate object area, but need Classify to each object candidate region, object set of candidate regions is huger under normal circumstances, so classification needs very The long time;And well-designed post-processing technology is needed to screen classified object area, such as non local very big Value inhibition etc., in addition these technologies are still deposited in the identification of complex scene object detection all without the global information for utilizing image In defect.
Summary of the invention
The purpose of the present invention is to provide a kind of quick object identification method of the mobile robot based on deep learning,
Operational efficiency declines caused by for being separated due to detection with identification component in object detection identifying system, synchronous It is stronger to introduce ability to express in order not to lose the accuracy rate of detection for the function of completing object detection prediction and identify its classification The multilayer depth network of residual error formula improves the accuracy of integrated form scheme, solves shortcoming of the existing technology.
The present invention adopts the following technical scheme that realization:
1, the quick object identification method of a kind of mobile robot based on deep learning, which is characterized in that including walking as follows It is rapid:
1) mobile picture obtains: the vision data perceived in moving process by camera to obtain robot, deep Picture is spent because can be used to restore scene physical system to establish the constraint condition in scene it includes depth information.
2) image data pre-processes: two software units: color picture pretreatment unit and depth map are used in this step Piece pretreatment unit, color picture pretreatment unit is to carry out piecemeal processing to the color picture of input, by picture in its entirety point It is segmented into Pork-pieces grid, by one of its Marking the cell object thus if the center for occurring object in grid block therein Point, and corresponding to this grid, neural network will predict that multiple objects surround frame, and for the object category of this grid ownership It carries out certain confidence level to generate, provides corresponding data basis for object identification;
Depth picture pretreatment unit to generate the constraint condition under indoor environment, by the processing of data of adjusting the distance and Gravity vector is estimated, to establish corresponding physical environment system, corresponding environment is established about according to this environmental system Beam, such as the reference obtained between object and plane are measured, and eliminate mistake using this environmental constraints with identification module in detection Solution;
3) picture feature is extracted: being completed by building multilayer residual error neural network, is mainly included two software units: volume Product core feature extraction unit and residual unit, then using convolution kernel feature extraction unit and residual unit as the group of neural network Laminated structure is carried out at part, so that forming depth characteristic extracts neural network, completes the multilayer distributed reproduction of characteristics of image;
A) convolution kernel feature extraction: given piece image, system is by reusing a variety of convolution kernel functions, block normalization The mode of unit and non-linear correction unit extracts the characteristics of image of suitable object detection identification, and wherein convolution kernel function is one A small data window, such as the data window of 3*3, convolution operation are that data window slides on global picture region, and And seek product with response picture position by each in data window and sum by data window region to result, accordingly Convolution kernel function and network parameter will be obtained by training, and non-linear unit is by increasing the non-linear expression come Enhanced feature Ability.Picture local feature surrounds the prediction that enters data to complete as neural network at the generation of frame and object category Reason.
B) residual unit: the phenomenon that degeneration also will appear while exporting convergence due to the depth structure of neural network: I.e. with the increase of network depth, although precision is gradually increased, decline rapidly suddenly when reaching an extreme value.And this Phenomenon can not make alleviation not due to over-fitting causes, by continuing growing network depth to this.Compared to traditional scheme Using the output of previous feature extraction unit (specific combination of convolution kernel, non-linear unit) as the suitable of next layer of feature unit Sequence model) output of previous feature unit is added with input as the defeated of next layer of feature extraction unit by residual error neural unit Enter;
4) picture prediction exports:
Any given input picture, algorithm output result include in picture potential object encirclement frame size, The classification and confidence level of coordinate, object.Due to that may include multiple objects in a secondary picture, in order to reduce the uncertain of model Property, want to predict than in surrounding frame from whole picture prediction single body, the present invention attempts the grid that picture is first divided into p*p It whether there is object in each grid, if there is coordinate, size, classification and the confidence level (6 dimension) of prediction object.Because same One object possibly is present in multiple grids, and the cooperation prediction of multiple grids increases the robustness of object prediction and accurate Rate.And corresponding to each grid, neural network will predict that k possible objects surround frame, because may be same in a grid When there are the region of multiple objects, (p*p*k*6) a output is predicted using linear classifier, is finally deleted using non-maximum restraining Except redundancy and object window that confidence level is small.
5) environmental constraints optimize: environmental constraints optimization module is to generate corresponding environmental constraints item according to depth data Part can be calculated in environment since depth data can restore the three-dimensional point cloud structure in space by point cloud data Gravity vector, to be described to form constraint condition to plane and object come bottom-up according to gravity vector;
6) picture recognition exports: after environmental constraints optimization, system will be after neural network detection identification And the result after being optimized by environmental constraints is exported to robot receiving module and is handled, picture recognition output module is defeated simultaneously Go out for object detection with identification as a result, the central point x-axis y-axis of output data inclusion body classification, encirclement detection block is sat Mark surrounds the data such as detection width of frame and height.This data is used to robot assisted environment understanding and navigation composition.Having Under the premise of having high accuracy, whole system be can reach as the movement of robot handles (> 24FPS) in real time, to guarantee The timeliness and accuracy of robot command adapted thereto.
2, the quick object identification method of the mobile robot according to claim 1 based on deep learning, feature Be, the camera be depth camera, depth camera can also be obtained while obtaining ordinary color picture with away from From the depth picture based on information.
The method have the benefit that:
1) by the unified integration of detection and recognition result, traditional object identification system is overcome to need to add using object identification The complexity and unstability of detection;2) generation for passing through multilayer residual error network design and surrounding gravity constraint condition, overcomes collection The shortcomings that accepted way of doing sth object identification system poor accuracy;3) integration for detecting identification mission also ensures the treatment effeciency of system, mentions Sensing capability in high robot moving process.
Detailed description of the invention
Fig. 1 is the overall flow figure of recognition methods of the present invention.
Fig. 2 is multilayer residual error neural network diagram.
Fig. 3 is residual error neural unit figure.
Specific embodiment
By the following description of the embodiment, the public understanding present invention will more be facilitated, but can't should be by Shen Given specific embodiment of asking someone is considered as the limitation to technical solution of the present invention, the definition of any pair of component or technical characteristic Be changed and/or to overall structure make form and immaterial transformation is regarded as defined by technical solution of the present invention Protection scope.
Step 1, mobile picture obtain: mobile picture obtains module acquisition robot and passes through camera sense in moving process The vision data known, vision data include depth picture and common RGB color picture two parts.
Step 2, image data pretreatment: image data preprocessing module obtains the picture obtained in module to picture and carries out Pretreatment, firstly generates corresponding color picture and depth picture, rear color picture pretreatment unit is to the color to input Picture carries out piecemeal, and picture in its entirety is divided into one piece of block grid, positions object using the mode of grid.Depth picture is located in advance Reason unit is iterated calculating to gravity vector, first calculating picture plane normal vector.Therewith by horizontal with vertical normal vector into Row Classified optimization iteration, the final accurate approximation for obtaining gravity vector establish bottom-up physical environment constraint.
Step 3, picture feature are extracted: picture feature extraction module uses the shape of multilayer residual error neural network combination convolution kernel Formula highlights feature from level to level, as shown below: with going deep into for neural network, it is possible to find the feature of object edge is got over It is more obvious, light tone is presented.Edge Gradient Feature is carried out feature again and the feature for forming higher is combined to present by neural network, from And complete prediction.
Step 4, picture prediction output: picture prediction output module is carried out surrounding frame size, be sat to object in input picture The prediction of mark, the classification of object and confidence level.The present invention generates multiple prediction blocks, reuse non-maximum restraining delete redundancy and And the multiple prediction results of object window generation that confidence level is small.
Step 5, environmental constraints optimization: environmental constraints optimization module utilizes gravity vector direction, carries out to plane and object Height description forms constraint condition.Such as people is higher than seat plane, object is higher than desk plane etc., for prediction network output As a result it can be optimized according to constraint, to remove the mistake solution under some environmental constraints, the encirclement frame of human body also occurs In prediction result, it is judged as correct through overconstrained condition.
Step 6, picture recognition output: picture recognition output module output recognition result includes classification, the center for surrounding frame Point x-axis y-axis coordinate surrounds the data such as width of frame and height, uses to robot assisted environment understanding and navigation composition.Having Under the premise of having high accuracy, whole system be can reach as the movement of robot handles (> 24FPS) in real time, to guarantee The timeliness and accuracy of robot command adapted thereto.
Certainly, the present invention can also have other various embodiments, without deviating from the spirit and substance of the present invention, Those skilled in the art can make various corresponding changes and modifications according to the present invention, but these it is corresponding change and Deformation all should fall within the scope of protection of the appended claims of the present invention.

Claims (2)

1. a kind of quick object identification method of mobile robot based on deep learning, which comprises the steps of:
1) mobile picture obtains: the vision data perceived in moving process by camera to obtain robot;
2) image data pre-processes: two software units are used in this step: color picture pretreatment unit and depth picture are pre- Picture in its entirety is divided by processing unit, color picture pretreatment unit to carry out piecemeal processing to the color picture of input Pork-pieces grid, by a part of its Marking the cell object thus if the center for occurring object in grid block therein, And correspond to this grid, neural network will predict multiple objects surround frame, and for this grid ownership object category into The certain confidence level of row generates, and provides corresponding data basis for object identification;
Processing and counterweight of the depth picture pretreatment unit to generate the constraint condition under indoor environment, by data of adjusting the distance Force vector is estimated, to establish corresponding physical environment system, establishes corresponding environmental constraints according to this environmental system;
3) picture feature is extracted: being completed by building multilayer residual error neural network, is mainly included two software units: convolution kernel Feature extraction unit and residual unit, then using convolution kernel feature extraction unit and residual unit as the composition portion of neural network Divide and carry out laminated structure, so that forming depth characteristic extracts neural network, completes the multilayer distributed reproduction of characteristics of image;
A) convolution kernel feature extraction: given piece image, system is by reusing a variety of convolution kernel functions, block normalization unit The characteristics of image of suitable object detection identification is extracted with the mode of non-linear correction unit, wherein convolution kernel function is a number According to window, convolution operation is that data window slides on global picture region, and presses each in data window and sound Picture position is answered to seek product and sum by data window region to result, corresponding convolution kernel function and network parameter will pass through To obtain, non-linear unit will be as mind by increasing the non-linear ability to express come Enhanced feature, picture local feature for training The generation processing for entering data to complete prediction and surrounding frame and object category through network;
B) residual unit: the output of previous feature unit is added with input as next layer of feature extraction list by residual error neural unit The input of member;
4) picture prediction exports: predicting output using linear classifier, the object of redundancy is finally deleted using non-maximum restraining Window;
5) environmental constraints optimize: environmental constraints optimization module is to generate corresponding environmental constraint according to depth data;
6) picture recognition export: by environmental constraints optimization after, system will by neural network detection identification after and lead to Result after crossing environmental constraints optimization, which is exported to robot receiving module, to be handled.
2. the quick object identification method of the mobile robot according to claim 1 based on deep learning, which is characterized in that The camera is depth camera, and depth camera can also obtain while obtaining ordinary color picture with range information Based on depth picture.
CN201610581928.2A 2016-07-22 2016-07-22 A kind of quick object identification method of mobile robot based on deep learning Active CN106228162B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610581928.2A CN106228162B (en) 2016-07-22 2016-07-22 A kind of quick object identification method of mobile robot based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610581928.2A CN106228162B (en) 2016-07-22 2016-07-22 A kind of quick object identification method of mobile robot based on deep learning

Publications (2)

Publication Number Publication Date
CN106228162A CN106228162A (en) 2016-12-14
CN106228162B true CN106228162B (en) 2019-05-17

Family

ID=57531308

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610581928.2A Active CN106228162B (en) 2016-07-22 2016-07-22 A kind of quick object identification method of mobile robot based on deep learning

Country Status (1)

Country Link
CN (1) CN106228162B (en)

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106919897B (en) * 2016-12-30 2020-05-22 华北电力大学(保定) Human face image age estimation method based on three-level residual error network
US10275687B2 (en) * 2017-02-16 2019-04-30 International Business Machines Corporation Image recognition with filtering of image classification output distribution
CN108509820B (en) * 2017-02-23 2021-12-24 百度在线网络技术(北京)有限公司 Obstacle segmentation method and device, computer equipment and readable medium
CN107123111B (en) * 2017-04-14 2020-01-24 惠州旭鑫智能技术有限公司 Deep residual error network construction method for mobile phone screen defect detection
KR20180134230A (en) * 2017-06-08 2018-12-18 삼성전자주식회사 Cleaning robot and controlling method of thereof
WO2018232754A1 (en) * 2017-06-23 2018-12-27 Microsoft Technology Licensing, Llc. Joint object detection based on collaborative information
CN108229296B (en) * 2017-09-30 2021-04-02 深圳市商汤科技有限公司 Face skin attribute identification method and device, electronic equipment and storage medium
CN108986022A (en) 2017-10-30 2018-12-11 上海寒武纪信息科技有限公司 Image beautification method and related product
CN107729885B (en) * 2017-11-23 2020-12-29 中电科新型智慧城市研究院有限公司 Face enhancement method based on multiple residual error learning
CN107967773A (en) * 2017-12-01 2018-04-27 旗瀚科技有限公司 A kind of supermarket self-help purchase method of view-based access control model identification
CN108229548A (en) * 2017-12-27 2018-06-29 华为技术有限公司 A kind of object detecting method and device
CN108279671B (en) * 2018-01-08 2021-12-14 深圳市易成自动驾驶技术有限公司 Terahertz-based environment sensing method and device and computer readable storage medium
CN110675324B (en) * 2018-07-02 2023-10-10 上海寰声智能科技有限公司 4K ultra-high definition image sharpening processing method
CN109658418A (en) * 2018-10-31 2019-04-19 百度在线网络技术(北京)有限公司 Learning method, device and the electronic equipment of scene structure
JP2022520019A (en) * 2019-02-15 2022-03-28 エスゼット ディージェイアイ テクノロジー カンパニー リミテッド Image processing methods, equipment, mobile platforms, programs
CN110163260B (en) * 2019-04-26 2024-05-28 平安科技(深圳)有限公司 Residual network-based image identification method, device, equipment and storage medium
CN110948492B (en) * 2019-12-23 2021-10-22 浙江大学 Three-dimensional grabbing platform and grabbing method based on deep learning
GB2609708B (en) * 2021-05-25 2023-10-25 Samsung Electronics Co Ltd Method and apparatus for video recognition
CN118297945B (en) * 2024-06-05 2024-08-13 江西师范大学 Defect detection method and system based on position constraint residual error and sliding window aggregation

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103218831A (en) * 2013-04-21 2013-07-24 北京航空航天大学 Video moving target classification and identification method based on outline constraint
CN104408469A (en) * 2014-11-28 2015-03-11 武汉大学 Firework identification method and firework identification system based on deep learning of image
CN105260749A (en) * 2015-11-02 2016-01-20 中国电子科技集团公司第二十八研究所 Real-time target detection method based on oriented gradient two-value mode and soft cascade SVM
CN105740758A (en) * 2015-12-31 2016-07-06 上海极链网络科技有限公司 Internet video face recognition method based on deep learning

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103218831A (en) * 2013-04-21 2013-07-24 北京航空航天大学 Video moving target classification and identification method based on outline constraint
CN104408469A (en) * 2014-11-28 2015-03-11 武汉大学 Firework identification method and firework identification system based on deep learning of image
CN105260749A (en) * 2015-11-02 2016-01-20 中国电子科技集团公司第二十八研究所 Real-time target detection method based on oriented gradient two-value mode and soft cascade SVM
CN105740758A (en) * 2015-12-31 2016-07-06 上海极链网络科技有限公司 Internet video face recognition method based on deep learning

Also Published As

Publication number Publication date
CN106228162A (en) 2016-12-14

Similar Documents

Publication Publication Date Title
CN106228162B (en) A kind of quick object identification method of mobile robot based on deep learning
CN107622244B (en) Indoor scene fine analysis method based on depth map
CN110163110B (en) Pedestrian re-recognition method based on transfer learning and depth feature fusion
CN109584248B (en) Infrared target instance segmentation method based on feature fusion and dense connection network
CN112288008B (en) Mosaic multispectral image disguised target detection method based on deep learning
CN104115192B (en) Three-dimensional closely interactive improvement or associated improvement
CN103971386B (en) A kind of foreground detection method under dynamic background scene
CN107292234B (en) Indoor scene layout estimation method based on information edge and multi-modal features
CN114782691A (en) Robot target identification and motion detection method based on deep learning, storage medium and equipment
CN111611874B (en) Face mask wearing detection method based on ResNet and Canny
CN106709568A (en) RGB-D image object detection and semantic segmentation method based on deep convolution network
CN108305260B (en) Method, device and equipment for detecting angular points in image
CN114758288A (en) Power distribution network engineering safety control detection method and device
CN106408030A (en) SAR image classification method based on middle lamella semantic attribute and convolution neural network
CN112070010B (en) Pedestrian re-recognition method for enhancing local feature learning by combining multiple-loss dynamic training strategies
CN106910202B (en) Image segmentation method and system for ground object of remote sensing image
CN105303163B (en) A kind of method and detection device of target detection
CN106991411B (en) Remote Sensing Target based on depth shape priori refines extracting method
CN114519819B (en) Remote sensing image target detection method based on global context awareness
CN103105924A (en) Man-machine interaction method and device
CN106503170B (en) It is a kind of based on the image base construction method for blocking dimension
CN105809716A (en) Superpixel and three-dimensional self-organizing background subtraction algorithm-combined foreground extraction method
CN110287798A (en) Vector network pedestrian detection method based on characteristic module and context fusion
CN115797736A (en) Method, device, equipment and medium for training target detection model and target detection
CN110245602A (en) A kind of underwater quiet target identification method based on depth convolution feature

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant
TR01 Transfer of patent right

Effective date of registration: 20240715

Address after: 9th Floor, Building C2, No. 88 Dazhou Road, Yuhuatai District, Nanjing City, Jiangsu Province, China 210012

Patentee after: Nanjing Fudian Culture Communication Co.,Ltd.

Country or region after: China

Address before: Room 1002, Unit 2, Building 16, Huangce Jiayuan, No. 19 Shuangtang Road, Qinhuai District, Nanjing City, Jiangsu Province 210000

Patentee before: Wang Wei

Country or region before: China

TR01 Transfer of patent right