CN107885327B - Fingertip detection method based on Kinect depth information - Google Patents
Fingertip detection method based on Kinect depth information Download PDFInfo
- Publication number
- CN107885327B CN107885327B CN201711021681.XA CN201711021681A CN107885327B CN 107885327 B CN107885327 B CN 107885327B CN 201711021681 A CN201711021681 A CN 201711021681A CN 107885327 B CN107885327 B CN 107885327B
- Authority
- CN
- China
- Prior art keywords
- hand
- point
- curvature
- points
- kinect
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 14
- 238000000034 method Methods 0.000 claims abstract description 16
- 210000000707 wrist Anatomy 0.000 claims abstract description 9
- 230000002146 bilateral effect Effects 0.000 claims abstract description 7
- 238000001914 filtration Methods 0.000 claims abstract description 7
- 210000004247 hand Anatomy 0.000 claims abstract description 5
- 238000007781 pre-processing Methods 0.000 claims abstract description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 230000000694 effects Effects 0.000 claims description 2
- 238000012216 screening Methods 0.000 claims 1
- 230000009466 transformation Effects 0.000 claims 1
- 230000003993 interaction Effects 0.000 abstract description 11
- 238000000605 extraction Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/012—Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a fingertip detection method based on Kinect depth information, wherein a device Kinect is connected with a computer through a cable; the method is characterized by comprising the following specific steps: step 1, extracting hands and acquiring palm coordinates: step 2, fingertip positioning: including image preprocessing and hand contouring; performing combined bilateral filtering on the extracted hand region; and (3) approximating the appointed point set by using a Douglas-Puck algorithm, finding out a polygonal fitting curve of the contour and drawing a fitting curve of the hand. And 3, searching the steps by using a covexHull () function, and analyzing to obtain convex wrap points of the hand. And 4, calculating the curvature of the obtained convex hull point, and setting a proper threshold value to remove the convex hull point at the wrist according to the difference between the curvature of the wrist and the curvature of the finger tip. The method can complete the gesture recognition task accurately in real time, improves the real-time performance and accuracy of Kinect gesture recognition, and improves natural gesture interaction experience.
Description
Technical Field
The invention relates to a fingertip detection method based on depth information, and belongs to the technical field of human-computer interaction.
Background
With the development of the human-computer interaction technology, a natural interaction mode becomes the development direction of the human-computer interaction technology. In recent years, natural and intuitive interaction is performed by human hands without wearing auxiliary equipment, and the method becomes a research hotspot in the field.
The key to achieving natural human hand interaction is accurate recognition of gestures. Currently, gesture recognition methods that do not rely on wearable aids include mainly color camera-based and depth sensor-based methods. The gesture recognition based on the color camera is easily affected by the complex lighting background and other situations, and particularly, the recognition accuracy is extremely low under the situation that the background is very complex. More and more researchers are beginning to utilize depth sensors to perform gesture recognition to acquire information of an object in a three-dimensional space, so as to improve accuracy of human hand detection and tracking. With the advent of Kinect, Kinect provides a new development direction for gesture recognition. Many scholars begin to use Kinect to perform gesture recognition, however, most of the existing methods are complicated in steps, complex in algorithm and poor in real-time performance.
Disclosure of Invention
The invention aims to provide a fingertip detection method based on Kinect depth information, which can accurately complete gesture recognition tasks in real time, improve the real-time performance and accuracy of Kinect gesture recognition, improve natural gesture interaction experience,
the technical scheme of the invention is realized as follows: a fingertip detection method based on Kinect depth information is characterized in that a device Kinect is connected with a computer through a cable; the method is characterized by comprising the following specific steps:
1) and carrying out joint bilateral filtering on the extracted hand region. The depth image and the color image of the same frame are collected through Kinect, and then the spatial distance weight of the depth image and the gray scale weight of the color image are calculated respectively by utilizing a Gaussian kernel function. And multiplying the spatial distance weight by the gray weight to obtain a combined filtering weight, and replacing a Gaussian kernel function with the fast Gaussian transform to generate the combined bilateral filter. And finally, performing convolution operation on the result filtered by the filter and the extracted hand region image, thereby realizing the effect of smooth edge protection of the finger tip region.
2) Selecting proper threshold value for the filtered image, carrying out binarization to generate a binary single-channel image, and then applying a findContours () function to search the outlines C of all feature points in the binarized imageiEach contour is a polygon and is composed of a pixel sequence, and the obtained hand feature point contour set C is represented as follows:
C＝findContours(Ht(x,y))＝{Ci} (1)
and drawing the searched outline by using a drawContours () function.
3) And (3) approximating the appointed point set by using a Douglas-Puck algorithm, finding out a polygonal fitting curve of the contour and drawing a fitting curve of the hand. Virtually connecting a straight line to the first and last points of each curve, calculating the distance between all the points and the straight line, and finding out the maximum distance value dmaxBy dmaxCompared with the tolerance D: if d ismax< D, the middle points on this curve are all rounded off; if d ismaxNot less than D, reserve DmaxAnd dividing the curve into two parts by taking the corresponding coordinate point as a boundary, and repeatedly using the method for the two parts.
Step 3, utilizing the covexHull () function to search the obtained resultMaximum area value of the profile CmaxAnd analyzing to obtain the convex hull points of the hands.
And 4, calculating the curvature of the obtained convex hull point, and setting a proper threshold value to remove the convex hull point at the wrist according to the difference between the curvature of the wrist and the curvature of the finger tip.
1) Taking a point on the hand contour, taking the first M points q of the point p on the contour and the M-th point r after the point p, the curvature of the point p can be used as a vector
And
is represented by the cosine of the angle alpha. The value of the p-point curvature can be expressed as follows:
2) a threshold value L is adopted to judge the finger-like cusp, the pixel points with the curvature larger than or equal to L are screened out,
as a finger tip.
And 5: the curves at the two sides of the finger can be approximate to a group of parallel lines, 10 pixel points are taken forward and 10 pixel points are taken backward for the p points meeting the conditions obtained in the step 4, the included angle of the p points is obtained according to the formula (2), and a proper threshold range is adopted, so that the condition that the finger is mistakenly judged as the fingertip point when the finger is bent is eliminated.
The fingertip detection based on the Kinect depth information can be completed through the steps.
Compared with the fingertip detection at the present stage, the method has the advantages that:
1. the fingertip detection method simplifies the process of hand extraction.
2. The method can more accurately acquire the hand contour fitting curve.
3. The invention can well filter out non-fingertip points.
The main purpose is to realize the real-time and accurate positioning of the finger tip; the human-computer interaction realized by the traditional camera is improved, the accurate positioning of the fingertips ensures that the interaction accuracy of the hand and the fingertips of the virtual object is better improved, and the natural human-computer interaction experience is improved.
Drawings
FIG. 1 is a schematic view of an initial hand gesture and apparatus.
Fig. 2 is a flowchart of the steps of the fingertip detection method based on depth images according to the present invention.
Detailed Description
The embodiments of the present invention will be described in detail below with reference to the accompanying drawings and specific examples.
Step S11, placing the Kinect on the desktop, with the palm perpendicular to the desktop and the palm facing the Kinect, about one meter away from the Kinect, as shown in fig. 1. And the palm is pushed forwards and then retracted, and the gesture tracking function of the NITE function library is triggered.
Step S12 is to acquire the coordinates of the palm using the NITE function library, and then calculate the approximate depth range of the hand from the palm point depth.
Step S13, setting a search area and a depth threshold value using the depth range obtained in step S12, and separating the hand image from the background by multiplying n rows and n columns of elements by the palm area through an n × n matrix using a depth binary mask.
Step S21, first, a depth image and a color image of the same frame are collected by the Kinect1, and then a spatial distance weight of the depth image and a gray scale weight of the color image are respectively calculated by using a gaussian kernel function. And multiplying the spatial distance weight by the gray weight to obtain a combined filtering weight, and replacing a Gaussian kernel function with the fast Gaussian transform to generate the combined bilateral filter.
In step S22, the hand region image extracted in S13 is convolved with the filtering result of the filter designed in S21. Thereby obtaining a joint bilateral filtered image.
And step S23, selecting proper threshold values for the image obtained in step S22, and carrying out binarization to generate a binary single-channel image.
Step S24, applying findContours () function to retrieve binary valuesContour C of all feature points in the normalized imageiEach contour is a polygon and is composed of a sequence of pixels, and expression (1) is as follows.
C＝findContours(Ht(x,y))＝{Ci} (1)
And step S25, drawing the searched outline in the step S24 by using a drawContours () function.
Step S26, virtually connecting a straight line to the head and tail points of each curve drawn in the step S25, calculating the distance between all the points and the straight line, and finding out the maximum distance value dmaxBy dmaxCompared with the tolerance D: if d ismax< D, the middle points on this curve are all rounded off; if d ismaxNot less than D, reserve DmaxAnd dividing the curve into two parts by taking the corresponding coordinate point as a boundary, and repeatedly using the method for the two parts. Finding out a polygonal fitting curve of the outline and drawing a fitting curve of the hand.
And step S3, performing contour analysis on the hand fitting curve obtained in the step S26 by using a convexHull () function to obtain a convex wrapping point of the hand.
And step S4, calculating the curvature of the obtained convex hull point, and setting a proper threshold value to remove the convex hull point at the wrist according to the difference between the curvature of the wrist and the curvature of the fingertip.
Step S41, taking a point p on the contour of the hand, taking M points q of the point p on the contour and M points r subsequent to the point p, then the curvature of the point p can be used as a vector
And
is represented by the cosine of the angle alpha. The value of the p-point curvature can be expressed as follows.
And step S42, adopting a threshold value L to judge that the finger-like fingertip points screen out pixel points P with the curvature being more than or equal to L, and removing the convex hull of the wrist through curvature detection.
Step S5: the curves at the two sides of the finger can be approximate to a group of parallel lines, 10 pixel points are taken forward from the point P meeting the condition obtained in the step S42, 10 pixel points are taken backward, the included angle of the point P is obtained according to the formula (2), and a proper threshold range is adopted, so that the condition that the finger is mistakenly judged as the fingertip point when the finger is bent is eliminated. Although illustrative embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be understood that the present invention is not limited to the scope of the embodiments, and various changes may be made apparent to those skilled in the art as long as they are within the spirit and scope of the present invention as defined and defined by the appended claims, and all matters of the invention which utilize the inventive concepts are protected.
Claims (1)
1. A fingertip detection method based on Kinect depth information is characterized in that a device Kinect is connected with a computer through a cable; the method is characterized by comprising the following specific steps:
step 1, extracting hands and acquiring palm coordinates: acquiring human hand information by using a Kinect, acquiring a hand center coordinate by using an NITE function library, calculating an approximate depth range of a hand by the depth of a hand center point, setting a search region and a depth threshold, multiplying n rows and n columns of elements by a hand center region by using a depth binary mask through an n x n matrix, and separating a hand image from a background;
step 2, fingertip positioning: the method comprises the steps of image preprocessing and hand contour acquisition;
1) performing combined bilateral filtering on the extracted hand region; firstly, acquiring a depth image and a color image of the same frame through a Kinect, and then respectively calculating a spatial distance weight of the depth image and a gray scale weight of the color image by utilizing a Gaussian kernel function; multiplying the spatial distance weight by the gray weight to obtain a combined filtering weight, and replacing a Gaussian kernel function with rapid Gaussian transformation to generate a combined bilateral filter; finally, performing convolution operation on the result filtered by the filter and the extracted hand region image so as to realize the effect of smooth edge protection of the finger tip region;
2) selecting a proper threshold value for binarization of the filtered image to generate a binary single-channel image Ht(x, y), wherein x and y are pixel coordinates, and then a findContours () function is applied to retrieve the outlines C of all the feature points in the binarized imageiEach contour is a polygon and is composed of a pixel sequence, and the obtained hand feature point contour set C is represented as follows:
C＝findContours(Ht(x,y))＝{Ci} (1)
drawing the searched outline by using a drawContours () function;
3) approximating the contour set C of the hand characteristic points by using a Douglas-Puck algorithm, finding out a polygonal fitting curve of the contour and drawing a fitting curve of the hand; virtually connecting a straight line to the first and last points of each curve, calculating the distance between all the points in the curve and the straight line, and finding out the maximum distance value dmaxBy dmaxCompared with the tolerance D: if d ismax< D, the middle points on this curve are all rounded off; if d ismaxNot less than D, reserve DmaxDividing the curve into two parts by taking the corresponding coordinate point as a boundary, and repeatedly using the method for the two parts;
step 3, searching the contour C of the maximum area value obtained in the step by using the covexHull () functionmaxThe convex hull of the hand is analyzed to obtain the convex hull point of the hand;
step 4, calculating the curvature of the obtained convex hull point, and setting a proper threshold value to remove the convex hull point at the wrist according to the difference between the curvature of the wrist and the curvature of the fingertip;
1) taking a point on the hand contour, taking the Mth point q before the point p on the contour and the Mth point r after the point p, the curvature of the point p can be used as a vector
And
is represented by the cosine of the angle alpha, the value of the curvature of the p pointCan be expressed as follows:
2) judging a finger tip point by adopting a threshold value L, and screening out pixel points with the curvature being more than or equal to L as the finger tip point;
and 5: the curves at the two sides of the finger can be approximate to a group of parallel lines, the fingertip points p screened out according to the curvature in the step 4 are subjected to forward 10 pixel points and backward 10 pixel points, the included angle of the p points is obtained according to a formula (2), and a proper threshold range is adopted, so that the condition that the finger is mistakenly judged as the fingertip point when the finger is bent is eliminated;
the fingertip detection based on the Kinect depth information can be completed through the steps.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711021681.XA CN107885327B (en) | 2017-10-27 | 2017-10-27 | Fingertip detection method based on Kinect depth information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711021681.XA CN107885327B (en) | 2017-10-27 | 2017-10-27 | Fingertip detection method based on Kinect depth information |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107885327A CN107885327A (en) | 2018-04-06 |
| CN107885327B true CN107885327B (en) | 2020-11-13 |
Family
ID=61782663
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711021681.XA Expired - Fee Related CN107885327B (en) | 2017-10-27 | 2017-10-27 | Fingertip detection method based on Kinect depth information |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107885327B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108805141A (en) * | 2018-05-29 | 2018-11-13 | 电子科技大学 | A kind of human skeleton artis location estimation method |
| CN109308741B (en) * | 2018-08-08 | 2023-04-07 | 长春理工大学 | Meta 2-based natural interaction handicraft creative design system |
| CN109344718B (en) * | 2018-09-03 | 2021-02-09 | 先临三维科技股份有限公司 | Fingertip identification method, device, storage medium and processor |
| CN110874179B (en) * | 2018-09-03 | 2021-09-14 | 京东方科技集团股份有限公司 | Fingertip detection method, fingertip detection device, and medium |
| CN109344793B (en) * | 2018-10-19 | 2021-03-16 | 北京百度网讯科技有限公司 | Method, apparatus, device and computer readable storage medium for recognizing handwriting in the air |
| CN109740497B (en) * | 2018-12-27 | 2022-07-26 | 河海大学 | Fingertip detection method based on least square curve fitting |
| CN111402368B (en) * | 2019-01-03 | 2023-04-11 | 福建天泉教育科技有限公司 | Correction method for drawing graph and terminal |
| CN110232321B (en) * | 2019-05-10 | 2021-07-06 | 奥比中光科技集团股份有限公司 | Method and device for detecting fingertip clicking position, terminal and computer storage medium |
| CN111354029B (en) * | 2020-02-26 | 2023-05-05 | 深圳市瑞立视多媒体科技有限公司 | Gesture depth determination method, device, equipment and storage medium |
| CN111950514B (en) * | 2020-08-26 | 2022-05-03 | 重庆邮电大学 | Depth camera-based aerial handwriting recognition system and method |
| CN113947683B (en) * | 2021-10-15 | 2022-07-08 | 兰州交通大学 | Fingertip point detection method and system and fingertip point motion track identification method and system |
| CN115908573B (en) * | 2023-02-20 | 2023-06-02 | 季华实验室 | Rubber glove opening positioning method, system, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103226387A (en) * | 2013-04-07 | 2013-07-31 | 华南理工大学 | Video fingertip positioning method based on Kinect |
| CN104899600A (en) * | 2015-05-28 | 2015-09-09 | 北京工业大学 | Depth map based hand feature point detection method |
| CN106295531A (en) * | 2016-08-01 | 2017-01-04 | 乐视控股（北京）有限公司 | A kind of gesture identification method and device and virtual reality terminal |
| CN106355598A (en) * | 2016-09-14 | 2017-01-25 | 南通大学 | Automatic wrist and finger joint motion degree measurement method |
| CN106503626A (en) * | 2016-09-29 | 2017-03-15 | 南京信息工程大学 | Being mated with finger contours based on depth image and refer to gesture identification method |
| WO2017134059A1 (en) * | 2016-02-05 | 2017-08-10 | Delphi Technologies, Inc. | System and method for detecting hand gestures in a 3d space |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016091457A (en) * | 2014-11-10 | 2016-05-23 | 富士通株式会社 | Input device, fingertip-position detection method, and computer program for fingertip-position detection |
-
2017
- 2017-10-27 CN CN201711021681.XA patent/CN107885327B/en not_active Expired - Fee Related
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103226387A (en) * | 2013-04-07 | 2013-07-31 | 华南理工大学 | Video fingertip positioning method based on Kinect |
| CN104899600A (en) * | 2015-05-28 | 2015-09-09 | 北京工业大学 | Depth map based hand feature point detection method |
| WO2017134059A1 (en) * | 2016-02-05 | 2017-08-10 | Delphi Technologies, Inc. | System and method for detecting hand gestures in a 3d space |
| CN106295531A (en) * | 2016-08-01 | 2017-01-04 | 乐视控股（北京）有限公司 | A kind of gesture identification method and device and virtual reality terminal |
| CN106355598A (en) * | 2016-09-14 | 2017-01-25 | 南通大学 | Automatic wrist and finger joint motion degree measurement method |
| CN106503626A (en) * | 2016-09-29 | 2017-03-15 | 南京信息工程大学 | Being mated with finger contours based on depth image and refer to gesture identification method |
Non-Patent Citations (2)
| Title |
|---|
| 《1 种基于 Kinect 深度图像的指尖检测与跟踪算法》;李智娴 等;《江苏农业科学》;20151231;第43卷(第5期);第416-419页 * |
| 《一种基于Kinect深度图的人像自动抠像算法》;李华 等;《长春理工大学学报（自然科学版）》;20161231;第39卷(第6期);第81-84页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107885327A (en) | 2018-04-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107885327B (en) | Fingertip detection method based on Kinect depth information | |
| US9916012B2 (en) | Image processing apparatus, image processing method, and program | |
| CN107563494B (en) | First-view-angle fingertip detection method based on convolutional neural network and heat map | |
| Shenoy et al. | Real-time Indian sign language (ISL) recognition | |
| US20130050076A1 (en) | Method of recognizing a control command based on finger motion and mobile device using the same | |
| EP3090382B1 (en) | Real-time 3d gesture recognition and tracking system for mobile devices | |
| CN103971102A (en) | Static gesture recognition method based on finger contour and decision-making trees | |
| CN103218605A (en) | Quick eye locating method based on integral projection and edge detection | |
| CN106503651B (en) | A kind of extracting method and system of images of gestures | |
| CN108171133A (en) | A kind of dynamic gesture identification method of feature based covariance matrix | |
| CN107357414B (en) | Click action recognition method and device | |
| CN112381045A (en) | Lightweight human body posture recognition method for mobile terminal equipment of Internet of things | |
| CN106503619B (en) | Gesture recognition method based on BP neural network | |
| Vivek Veeriah et al. | Robust hand gesture recognition algorithm for simple mouse control | |
| CN108876776B (en) | Classification model generation method, fundus image classification method and device | |
| Sokhib et al. | A combined method of skin-and depth-based hand gesture recognition. | |
| Pradhan et al. | A hand gesture recognition using feature extraction | |
| CN111651038A (en) | Gesture recognition control method based on ToF and control system thereof | |
| Yashas et al. | Hand gesture recognition: a survey | |
| KR101167784B1 (en) | A method for recognizing pointers and a method for recognizing control commands, based on finger motions on the back of the portable information terminal | |
| CN103077381A (en) | Monocular dynamic hand gesture recognition method on basis of fractional Fourier transformation | |
| Choi et al. | RGB-D camera-based hand shape recognition for human-robot interaction | |
| CN109359543B (en) | Portrait retrieval method and device based on skeletonization | |
| CN112084840A (en) | Finger vein identification method based on three-dimensional NMI | |
| Ying et al. | Fingertip detection and tracking using 2D and 3D information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20201113 |