[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN109359568A - A kind of human body critical point detection method based on figure convolutional network - Google Patents

A kind of human body critical point detection method based on figure convolutional network Download PDF

Info

Publication number
CN109359568A
CN109359568A CN201811154729.9A CN201811154729A CN109359568A CN 109359568 A CN109359568 A CN 109359568A CN 201811154729 A CN201811154729 A CN 201811154729A CN 109359568 A CN109359568 A CN 109359568A
Authority
CN
China
Prior art keywords
key point
human body
network
convolutional network
detection method
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201811154729.9A
Other languages
Chinese (zh)
Inventor
崔振
吕辉
许春燕
张桐
杨健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Seetatech Beijing Technology Co ltd
Original Assignee
Nanjing University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing University of Science and Technology filed Critical Nanjing University of Science and Technology
Priority to CN201811154729.9A priority Critical patent/CN109359568A/en
Publication of CN109359568A publication Critical patent/CN109359568A/en
Pending legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The human body critical point detection method based on figure convolutional network that the invention discloses a kind of extracts characteristics of human body using convolutional neural networks from image, predicts human body key point response diagram, determine initial key point coordinate;According to the key point coordinate of prediction, the local feature of the corresponding each key point of human body is extracted from convolutional neural networks;The graph model for establishing human body key point extracts the feature vector of each key point according to the positional relationship of each key point of human body and neighbouring key point;The feature vector of each key point of human body is inputted figure convolutional network, the offset of each key point is obtained, with initial key point coordinate plus offset to get the key point prediction result of optimization.The predicted key point of the present invention establishes graph model to human body key point, has better accounted for the connection between key point, improve the accuracy rate of crucial point prediction compared to other methods.

Description

A kind of human body critical point detection method based on figure convolutional network
Technical field
The present invention relates to image processing techniques, and in particular to a kind of human body critical point detection side based on figure convolutional network Method.
Background technique
Positioning human body key point is an important and challenging task in computer vision field, be can be used as The basis of many high-level semantic tasks (such as: action recognition, clothes parsing, human body identify again, human-computer interaction etc.).In recent years, With the introducing of convolutional neural networks, human body key point prediction field, which achieves, to be substantially in progress.But accurately to position human body There are still the situations of many difficulties for key point, for example key point is blocked, human body overlapping, view transformation.Newest some key points Detection method, such as the openpose project of CMU predict that more people are crucial by predicting human body key point and limbs vector simultaneously Point, achieves good results, but still solves problem above without proposing good method.
Summary of the invention
The human body critical point detection method based on figure convolutional network that the purpose of the present invention is to provide a kind of, compared to other Critical point detection method preferably considers the connection between each key point of human body, improves the standard of human body critical point detection True rate, overcome block, human body overlapping the problems such as.
The technical solution for realizing the aim of the invention is as follows: a kind of human body critical point detection side based on figure convolutional network Method includes the following steps:
Step 1 extracts characteristics of human body using convolutional neural networks from image, predicts human body key point response diagram, determines Initial key point coordinate;
Step 2, the key point coordinate according to prediction extract the office of the corresponding each key point of human body from convolutional neural networks Portion's feature;
Step 3, the graph model for establishing human body key point are closed according to the position of each key point of human body and neighbouring key point System, extracts the feature vector of each key point;
The feature vector of each key point of human body is inputted figure convolutional network by step 4, obtains the offset of each key point, With initial key point coordinate plus offset to get the key point prediction result of optimization.
Compared with prior art, the present invention its remarkable advantage are as follows: 1) present invention consider connection human body key point between, Graph model is established to human body key point, is blocking etc. in special circumstances, the position of shield portions key point can be predicted;2) originally Invention improves the accuracy rate of human body key point prediction by picture scroll product prediction optimization key point coordinate.
Detailed description of the invention
Fig. 1 is that the present invention is based on the flow charts of the human body critical point detection method of figure convolutional network.
Fig. 2 is the schematic diagram of the basic convolutional neural networks of critical point detection.
Fig. 3 is the schematic diagram of the graph model of human body key point.
Specific embodiment
In the following with reference to the drawings and specific embodiments, the present invention program is further illustrated.
As shown in Figure 1, the human body critical point detection method based on figure convolutional network, includes the following steps:
Step 1 extracts characteristics of human body using convolutional neural networks from image, predicts human body key point response diagram, determines Initial key point coordinate;
As a kind of specific embodiment, select ResNet network as basic convolutional neural networks, as shown in Fig. 2, mentioning Characteristics of human body is taken, deconvolution is reused or up-samples the feature return high-resolution that network is acquired, this can effectively improve The accuracy rate of neural network forecast key point.Corresponding each key point, network can export a corresponding response diagram, in every response diagram The middle position (location of pixels for being most likely to be corresponding key point) for choosing maximum value and maximum value, can be obtained initial position Coordinate (Xi,Yi) and confidence level Si, wherein i represents key point serial number.
The depth of ResNet network is deeper, and the ability for extracting feature is stronger, but with the increase of network depth, network mould The parameter of type can increase, and the training time can also greatly increase.As a kind of more specific embodiment, the present invention selects ResNet50 Network estimates the basic network of network as human posture.
Specific embodiment, which is had more, as one kind 2 warp laminations is connect, to obtain height after basic convolutional neural networks The key point response diagram of resolution ratio.
Step 2, the key point coordinate according to prediction extract the office of the corresponding each key point of human body from convolutional neural networks Portion's feature;
As a kind of specific embodiment, using RoIAlign method, key point pixel peripheral region in original image is extracted Feature, obtained key point local feature vectors Fi, for being initially not previously predicted out key point, filled up with ' 0 ', obtain key point Local feature vectors, Average Pool operation then is carried out to obtained each key point local feature vectors;
Step 3, the graph model for establishing human body key point are closed according to the position of each key point of human body and neighbouring key point System, extracts the feature vector of each key point;
According to the connection between each key point of human body, graph model is established, as shown in figure 3, neighbouring in order to better account for Positional relationship between key point, here to each key point, the key point and two adjacent to key point feature vector It is cascaded, forms the feature vector F' of key pointi
The feature vector of each key point of human body is inputted figure convolutional network by step 4, obtains the offset of each key point, With initial key point coordinate plus offset to get the key point prediction result of optimization.
Obtained feature vector F'iThe figure convolutional network of input prediction key point positional shift, as a kind of specific reality Mode is applied, the present invention uses 3 layers of figure convolutional network, preceding two layers of random initializtion, and it is 0 that initial value, which is arranged, in the last layer.Picture scroll Product network exports key point positional shift (Δ Xi,ΔYi), in addition the initial key point coordinate of basis key point prediction neural network forecast (Xi,Yi), the key point coordinate (X ' of available optimizationi,Y’i), formula is as follows:
Embodiment
In order to verify the validity of the present invention program, following emulation experiment is carried out.
Firstly, the response diagram generated according to the true coordinate of human body key point is provided as label, load networks ResNet50ImageNet pre-training model parameter is as initialization, training basis convolutional neural networks, then learns single picture Then the response diagram of the middle each key point of human body connects 2 warp laminations to obtain high-resolution key point response diagram.At every Position and the maximum value that maximum value is chosen in response diagram, can be obtained the initial position co-ordinates (X of corresponding key pointi,Yi) and set Reliability Si, i represents key point serial number.
Then, local feature is extracted from the last layer characteristic pattern of basic crucial point prediction network, used here as RoIAlign method extracts the feature of 20*20 key point pixel peripheral region in original image, and obtaining 7*7*C, (C indicates characteristic pattern Port number) key point local feature vectors Fi, then k obtained feature vector is operated using Average Pool, in turn, Obtain the feature vector of k 1*1*C dimension.
Then, in order to better account for the positional relationship between neighbouring key point, here to each key point, the key It puts and two is cascaded adjacent to the feature vector of key point, form the feature vector F' of 1*1*3C dimensioni.This example is set Setting key point number k is 15, and specific series system is following (current key point-is adjacent to the neighbouring key point 2 of key point 1 -):
(crown-nose-neck)
(nose-neck-crown)
(the right shoulder of the left shoulder-of neck -)
(the left left elbow-neck of shoulder -)
(the left left shoulder of elbow-left finesse -)
(the left shoulder of the left elbow-of left finesse -)
(the right right elbow-neck of shoulder -)
(the right right shoulder of elbow-right finesse -)
(the right shoulder of the right elbow-of right finesse -)
(the left right hip-left knee of hip -)
(the left hip of left knee-left ankle -)
(the left hip of left ankle-left knee -)
(the right knee of the right left hip-of hip -)
(the right right hip of knee-right ankle -)
(the right hip of the right knee-of right ankle -)
Finally, k obtained feature vector F'iThe figure convolutional network of input prediction key point positional shift, this example Using 3 layers of figure convolutional network, preceding two layers of random initializtion, it is 0 that initial value, which is arranged, in the last layer.The output of figure convolutional network is crucial Point positional shift (Δ Xi,ΔYi), in addition the initial key point coordinate (X of basis key point prediction neural network forecasti,Yi), it can obtain To the key point coordinate (X ' of optimizationi,Y’i), specific calculate uses formula 1.
Since human body key point is not key point isolated one by one, such as shoulder, elbow, wrist, these three are crucial Point is connected by limbs, is at every moment linked together.Using the method for the present invention, certain situations are such as blocked, even if wrist Key point can not be directly visible, by the location information of shoulder and elbow, can soon predict the position of wrist.
The complex situations of human body key point position in image are limited to, are such as blocked, overlapping etc., the key of convolutional neural networks Point prediction accuracy rate can not be promoted further, and the present invention passes through picture scroll product prediction optimization key point coordinate, it is possible to reduce or even gram The obstruction that some complex situations bring crucial point prediction is taken, the accuracy rate of human body key point prediction is improved.
In conclusion the present invention preferably considers the connection between each key point of human body, human body key point is improved The accuracy rate of detection.

Claims (6)

1. a kind of human body critical point detection method based on figure convolutional network, which comprises the following steps:
Step 1 extracts characteristics of human body using convolutional neural networks from image, predicts human body key point response diagram, determines initial Key point coordinate;
Step 2, the key point coordinate according to prediction, the part that the corresponding each key point of human body is extracted from convolutional neural networks are special Sign;
Step 3, the graph model for establishing human body key point, according to the positional relationship of each key point of human body and neighbouring key point, Extract the feature vector of each key point;
The feature vector of each key point of human body is inputted figure convolutional network by step 4, obtains the offset of each key point, with first Beginning key point coordinate is plus offset to get the key point prediction result of optimization.
2. the human body critical point detection method according to claim 1 based on figure convolutional network, which is characterized in that step 1 In, it selects ResNet network to extract characteristics of human body, reuses deconvolution or up-sample the feature return high score that network is acquired Resolution, corresponding each key point, network can export a corresponding response diagram, the position of maximum value chosen in every response diagram And maximum value is to get the initial position co-ordinates and confidence level of corresponding key point.
3. the human body critical point detection method according to claim 2 based on figure convolutional network, which is characterized in that step 1 In, select 50 network of ResNet to extract characteristics of human body, then connect 2 warp laminations to obtain high-resolution key point response diagram.
4. the human body critical point detection method according to claim 1 based on figure convolutional network, which is characterized in that step 2 In, it is special from the part for extracting key point corresponding region in convolutional neural networks in the last layer characteristic pattern using RoIAlign method Sign, obtains the local feature vectors of key point, for being initially not previously predicted out key point, is filled up with ' 0 ', obtain the office of key point Portion's feature vector.
5. the human body critical point detection method according to claim 1 based on figure convolutional network, which is characterized in that step 3 In, consider the positional relationship between neighbouring key point, to each key point, the key point and two adjacent to key point spy Sign vector is cascaded, and forms the feature vector of the key point.
6. the human body critical point detection method according to claim 1 based on figure convolutional network, which is characterized in that step 4 In, using 3 layers of figure convolutional network, wherein preceding two layers of random initializtion, it is 0 that initial value, which is arranged, in the last layer.
CN201811154729.9A 2018-09-30 2018-09-30 A kind of human body critical point detection method based on figure convolutional network Pending CN109359568A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201811154729.9A CN109359568A (en) 2018-09-30 2018-09-30 A kind of human body critical point detection method based on figure convolutional network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201811154729.9A CN109359568A (en) 2018-09-30 2018-09-30 A kind of human body critical point detection method based on figure convolutional network

Publications (1)

Publication Number Publication Date
CN109359568A true CN109359568A (en) 2019-02-19

Family

ID=65348500

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201811154729.9A Pending CN109359568A (en) 2018-09-30 2018-09-30 A kind of human body critical point detection method based on figure convolutional network

Country Status (1)

Country Link
CN (1) CN109359568A (en)

Cited By (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109887017A (en) * 2019-03-25 2019-06-14 北京奇艺世纪科技有限公司 A kind of similarity calculating method and device
CN110222685A (en) * 2019-05-16 2019-09-10 华中科技大学 One kind being based on two stage clothes key independent positioning method and system
CN110569819A (en) * 2019-09-16 2019-12-13 天津通卡智能网络科技股份有限公司 Bus passenger re-identification method
CN110688929A (en) * 2019-09-20 2020-01-14 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN111028212A (en) * 2019-12-02 2020-04-17 上海联影智能医疗科技有限公司 Key point detection method and device, computer equipment and storage medium
CN111444928A (en) * 2020-03-30 2020-07-24 北京市商汤科技开发有限公司 Key point detection method and device, electronic equipment and storage medium
CN111523387A (en) * 2020-03-24 2020-08-11 杭州易现先进科技有限公司 Method and device for detecting hand key points and computer device
CN111696130A (en) * 2019-03-12 2020-09-22 北京京东尚科信息技术有限公司 Target tracking method, target tracking apparatus, and computer-readable storage medium
CN111783457A (en) * 2020-07-28 2020-10-16 北京深睿博联科技有限责任公司 Semantic visual positioning method and device based on multi-modal graph convolutional network
CN112149477A (en) * 2019-06-28 2020-12-29 北京地平线机器人技术研发有限公司 Attitude estimation method, apparatus, medium, and device
CN112733767A (en) * 2021-01-15 2021-04-30 西安电子科技大学 Human body key point detection method and device, storage medium and terminal equipment
CN113034849A (en) * 2019-12-25 2021-06-25 海信集团有限公司 Infant nursing apparatus, nursing method and storage medium
CN113095254A (en) * 2021-04-20 2021-07-09 清华大学深圳国际研究生院 Method and system for positioning key points of human body part
WO2023185241A1 (en) * 2022-03-31 2023-10-05 腾讯科技(深圳)有限公司 Data processing method and apparatus, device and medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105787439A (en) * 2016-02-04 2016-07-20 广州新节奏智能科技有限公司 Depth image human body joint positioning method based on convolution nerve network
US20170213381A1 (en) * 2016-01-26 2017-07-27 Università della Svizzera italiana System and a method for learning features on geometric domains
CN107767419A (en) * 2017-11-07 2018-03-06 广州深域信息科技有限公司 A kind of skeleton critical point detection method and device
CN108062536A (en) * 2017-12-29 2018-05-22 纳恩博(北京)科技有限公司 A kind of detection method and device, computer storage media
CN108229355A (en) * 2017-12-22 2018-06-29 北京市商汤科技开发有限公司 Activity recognition method and apparatus, electronic equipment, computer storage media, program

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170213381A1 (en) * 2016-01-26 2017-07-27 Università della Svizzera italiana System and a method for learning features on geometric domains
CN105787439A (en) * 2016-02-04 2016-07-20 广州新节奏智能科技有限公司 Depth image human body joint positioning method based on convolution nerve network
CN107767419A (en) * 2017-11-07 2018-03-06 广州深域信息科技有限公司 A kind of skeleton critical point detection method and device
CN108229355A (en) * 2017-12-22 2018-06-29 北京市商汤科技开发有限公司 Activity recognition method and apparatus, electronic equipment, computer storage media, program
CN108062536A (en) * 2017-12-29 2018-05-22 纳恩博(北京)科技有限公司 A kind of detection method and device, computer storage media

Cited By (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111696130B (en) * 2019-03-12 2024-06-21 北京京东尚科信息技术有限公司 Target tracking method, target tracking device, and computer-readable storage medium
CN111696130A (en) * 2019-03-12 2020-09-22 北京京东尚科信息技术有限公司 Target tracking method, target tracking apparatus, and computer-readable storage medium
CN109887017A (en) * 2019-03-25 2019-06-14 北京奇艺世纪科技有限公司 A kind of similarity calculating method and device
CN109887017B (en) * 2019-03-25 2021-09-03 北京奇艺世纪科技有限公司 Similarity calculation method and device
CN110222685A (en) * 2019-05-16 2019-09-10 华中科技大学 One kind being based on two stage clothes key independent positioning method and system
CN112149477B (en) * 2019-06-28 2024-07-19 北京地平线机器人技术研发有限公司 Attitude estimation method, device, medium and equipment
CN112149477A (en) * 2019-06-28 2020-12-29 北京地平线机器人技术研发有限公司 Attitude estimation method, apparatus, medium, and device
CN110569819A (en) * 2019-09-16 2019-12-13 天津通卡智能网络科技股份有限公司 Bus passenger re-identification method
CN110688929A (en) * 2019-09-20 2020-01-14 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN110688929B (en) * 2019-09-20 2021-11-30 北京华捷艾米科技有限公司 Human skeleton joint point positioning method and device
CN111028212A (en) * 2019-12-02 2020-04-17 上海联影智能医疗科技有限公司 Key point detection method and device, computer equipment and storage medium
CN111028212B (en) * 2019-12-02 2024-02-27 上海联影智能医疗科技有限公司 Key point detection method, device, computer equipment and storage medium
CN113034849A (en) * 2019-12-25 2021-06-25 海信集团有限公司 Infant nursing apparatus, nursing method and storage medium
CN111523387A (en) * 2020-03-24 2020-08-11 杭州易现先进科技有限公司 Method and device for detecting hand key points and computer device
CN111523387B (en) * 2020-03-24 2024-04-19 杭州易现先进科技有限公司 Method and device for detecting key points of hands and computer device
CN111444928A (en) * 2020-03-30 2020-07-24 北京市商汤科技开发有限公司 Key point detection method and device, electronic equipment and storage medium
CN111783457A (en) * 2020-07-28 2020-10-16 北京深睿博联科技有限责任公司 Semantic visual positioning method and device based on multi-modal graph convolutional network
CN112733767A (en) * 2021-01-15 2021-04-30 西安电子科技大学 Human body key point detection method and device, storage medium and terminal equipment
CN113095254B (en) * 2021-04-20 2022-05-24 清华大学深圳国际研究生院 Method and system for positioning key points of human body part
CN113095254A (en) * 2021-04-20 2021-07-09 清华大学深圳国际研究生院 Method and system for positioning key points of human body part
WO2023185241A1 (en) * 2022-03-31 2023-10-05 腾讯科技(深圳)有限公司 Data processing method and apparatus, device and medium

Similar Documents

Publication Publication Date Title
CN109359568A (en) A kind of human body critical point detection method based on figure convolutional network
JP7101829B2 (en) Human body detection methods, devices, computer equipment and storage media
CN110211196B (en) Virtual fitting method and device based on posture guidance
EP3690702A1 (en) Motion recognition and gesture prediction method and device
US11270158B2 (en) Instance segmentation methods and apparatuses, electronic devices, programs, and media
KR20220004716A (en) Image processing method, apparatus, computer device and storage medium
CN105930767A (en) Human body skeleton-based action recognition method
CN109033946A (en) Merge the estimation method of human posture of directional diagram
CN108416266A (en) A kind of video behavior method for quickly identifying extracting moving target using light stream
CN112200818B (en) Dressing region segmentation and dressing replacement method, device and equipment based on image
Xin et al. Residual attribute attention network for face image super-resolution
CN109977912A (en) Video human critical point detection method, apparatus, computer equipment and storage medium
CN112036260B (en) Expression recognition method and system for multi-scale sub-block aggregation in natural environment
WO2020233427A1 (en) Method and apparatus for determining features of target
CN113608663B (en) Fingertip tracking method based on deep learning and K-curvature method
Liu et al. Hand Gesture Recognition Based on Single‐Shot Multibox Detector Deep Learning
Xu et al. CCFNet: Cross-complementary fusion network for RGB-D scene parsing of clothing images
CN109345504A (en) A kind of bottom-up more people's Attitude estimation methods constrained using bounding box
CN114170403A (en) Virtual fitting method, device, server and storage medium
CN104699243B (en) A kind of incorporeity virtual mouse method based on monocular vision
CN109583584A (en) The CNN with full articulamentum can be made to receive the method and system of indefinite shape input
CN115147508B (en) Training of clothing generation model and method and device for generating clothing image
CN112966600B (en) Self-adaptive multi-scale context aggregation method for crowded population counting
Li et al. Object detection on low-resolution images with two-stage enhancement
Hannuksela et al. Motion-based finger tracking for user interaction with mobile devices

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
TA01 Transfer of patent application right
TA01 Transfer of patent application right

Effective date of registration: 20210517

Address after: Room 550, scientific research complex building, Institute of computing, Chinese Academy of Sciences, no.6, South Road, Haidian District, Beijing, 100190

Applicant after: SEETATECH (BEIJING) TECHNOLOGY Co.,Ltd.

Address before: 210094 Xuanwu District, Jiangsu, Xiaolingwei 200, Nanjing

Applicant before: NANJING University OF SCIENCE AND TECHNOLOGY

RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20190219