CN102662470A - Method and system for implementation of eye operation - Google Patents

Abstract

The invention discloses a method and a system for implementation of eye operation. The distance between the iris center of a user and the canthus point of the user can be determined, and an operation instruction corresponding to the distance can be carried out according to a preset corresponding relation between the distance and the operation instruction. By the aid of the method and the system for implementation of eye operation, eye operation can be realized with no need of requiring the user to wear a helmet and the like, so that user experience is fine and the cost is low.

Description

A method and system for realizing eye operation

technical field

The present invention relates to the field of human-computer interaction technology, in particular to a method and system for realizing eye operation.

Background technique

With the rapid development of computer technology and human-computer interaction technology, new human-computer interaction devices and interaction methods have gradually become a research hotspot in the field of human-computer interaction, and a common feature of traditional computer input devices such as keyboards and mice is that they need People use their hands to operate, but for some users (such as disabled people with inconvenient physical activities), this will make them use traditional human-computer interaction equipment with certain limitations in terms of naturalness and friendliness, so Research on human-computer interaction equipment suitable for different objects has become the current development trend of human-computer interaction technology.

The visual system plays an important role in the human sensory system. It has been found through research that most of people's perception of external information is obtained from vision. If people's visual system is used to develop a human-computer interaction system, it will be very effective and Nature.

The human-computer interaction method based on eye-tracking technology has the characteristics of directness, friendliness and simplicity. The human-computer interaction technology based on eye-tracking technology is mainly to obtain the area of interest of the user according to the principle of people's eye movement, and develop a related interactive system to realize the control of the computer or peripheral equipment. The current gaze tracking technology is contact detection. Contact detection requires the user to wear a special helmet to detect eye movement information. If the helmet is worn for a long time, this will bring great trouble to the user, and the contact device is expensive.

Contents of the invention

In order to solve the above technical problems, the embodiment of the present invention provides a method and system for realizing eye operation, so as to realize eye operation without wearing a helmet. The technical solution is as follows:

A method of effecting eye manipulation comprising:

Determine the first position of the user's iris center and the second position of the user's eye corner;

Calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance;

According to the preset correspondence between the distance and the operation instruction, the first operation instruction corresponding to the first distance is executed.

Preferably, the step of determining the first position of the user's iris center and the second position of the user's eye corner includes:

Collect human body images;

identifying a face image in the human body image;

identifying eye images in the face image;

The first position of the user's iris center and the second position of the user's eye corner are determined according to the eye image.

Preferably, the step of identifying the face image in the human body image includes:

Under the YIQ chromaticity space, according to the distribution interval of the I channel component of the human face skin color, identify the human body skin color image in the human body image, the human body skin color image is framed by a rectangle, and the human body skin color image includes: a human face image and neck image;

The upper short side of the rectangle is kept fixed, the rectangle is shortened to become a square, and the image inside the square is recognized as a human face image.

Preferably, the step of identifying the face image in the human body image includes:

In the YUV chromaticity space, according to the distribution range of the phase angle θ of the hue distribution of the human face skin color, identify the human body skin color image in the human body image, the human body skin color image is framed by a rectangle, and the human body skin color image includes : face image and neck image;

The upper short side of the rectangle is kept fixed, the rectangle is shortened to become a square, and the image inside the square is recognized as a human face image.

Preferably, the step of identifying the eye image in the face image includes:

The face image framed by the square is divided into equal upper half and lower half;

dividing the upper half into equal left half and right half, wherein the image in the left half carries the image of the right eye region of the human eye, and the image in the right half carries the left eye region image of the human eye. eye area image;

The genetic algorithm is used to locate the image of the right eye region of the human eye from the left half of the image, and the genetic algorithm is used to locate the image of the left eye region of the human eye from the image of the right half.

Preferably, in the step of determining the first position of the center of the user's iris and the second position of the corner of the user's eye according to the eye image, the method for determining the first position includes:

determining at least three points on the edge of the iris according to the color difference between the iris and the sclera in the eyeball in the binary image;

The first position where the center of the iris is located is determined according to at least three points on the edge of the iris.

Preferably, in the step of determining the first position of the center of the user's iris and the second position of the corner of the user's eye according to the eye image, the method for determining the second position includes:

Estimate the second position by using the first position where the center of the iris is located to generate an estimated position;

The second position is determined by using the corner responsivity function in the improved Harris corner detection algorithm combined with the variance projection function to correct the estimated position.

Preferably, the step of calculating the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, wherein the distance is the first distance, includes:

According to the first position and the second position, multiple distances between the center of the user's iris and the corners of the user's eyes are calculated within a preset time period;

An average value of the plurality of distances is obtained, the average value being a first distance.

The present invention also provides a system for realizing eye operation, including: a position determination module, a distance determination module and an instruction execution module,

The position determination module is used to determine the first position where the user's iris center is located and the second position of the user's eye corner;

The distance determining module is used to calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance;

The instruction execution module is configured to execute the first operation instruction corresponding to the first distance according to the preset correspondence between the distance and the operation instruction.

Preferably, the position determining module includes: an image acquisition device, a face recognition module, a human eye recognition module, an iris center recognition module and an eye point recognition module,

The image acquisition device is used to acquire human body images;

The face recognition module is used to recognize the face image in the human body image;

The human eye recognition module is used to recognize the eye image in the human face image;

The iris center recognition module is used to determine the first position of the user's iris center according to the eye image;

The eye point recognition module is configured to determine the second position of the user's eye point according to the eye image.

By applying the above technical solutions, the present invention provides a method and system for realizing eye operations, which can determine the distance between the center of the user's iris and the corner of the user's eyes, and perform the same operation according to the correspondence between the preset distance and the operation instruction. Operation instructions corresponding to the distance. Since the present invention can realize the eye operation without the user wearing equipment such as a helmet, the user feels good and the cost is low.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments described in the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a schematic flowchart of a method for realizing eye operation provided by an embodiment of the present invention;

Fig. 2 is a schematic flowchart of another method for realizing eye operation provided by an embodiment of the present invention;

Fig. 3 is a schematic diagram of dividing an image by a method for realizing eye operation provided by an embodiment of the present invention;

Fig. 4 is a schematic flowchart of another method for realizing eye operation provided by an embodiment of the present invention;

Fig. 5 is a schematic flowchart of another method for realizing eye operation provided by an embodiment of the present invention;

FIG. 6 is a schematic flowchart of another method for implementing eye operations provided by an embodiment of the present invention;

Fig. 7 is a schematic flowchart of another method for realizing eye operation provided by an embodiment of the present invention;

Fig. 8 is a schematic flowchart of another method for realizing eye operation provided by an embodiment of the present invention;

FIG. 9 is a schematic structural diagram for realizing an eye operating system provided by an embodiment of the present invention;

FIG. 10 is a schematic structural diagram of another eye operating system provided by an embodiment of the present invention;

Fig. 11 is a schematic structural diagram of another eye operating system provided by an embodiment of the present invention;

Fig. 12 is a schematic structural diagram of another eye operating system provided by an embodiment of the present invention;

Fig. 13 is a schematic structural diagram of another implementation of the eye operating system provided by the embodiment of the present invention;

Fig. 14 is a schematic structural diagram of another eye operating system provided by an embodiment of the present invention;

Fig. 15 is a schematic structural diagram of another implementation of the eye operating system provided by the embodiment of the present invention;

FIG. 16 is a schematic structural diagram of another implementation of an eye operating system provided by an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts shall fall within the protection scope of the present invention.

As shown in Figure 1, a method for implementing eye operations provided by an embodiment of the present invention may include: Step S1, Step S2, and Step S3,

S1. Determine the first position of the center of the user's iris and the second position of the corner of the user's eye;

It is understandable that the iris belongs to the middle layer of the eyeball and is located at the front of the vascular membrane. The color of the iris is darker, and the color of the iris varies from person to person, such as blue or brown in Europe and the United States, and black or gray in China and other Asian countries. wait. The iris is located in front of the ciliary body and has the function of automatically adjusting the size of the pupil and how much light enters the eye. Located in the most anterior part of the vascular membrane, the pupil is located in the center of the iris. . The sclera is the white part of the outside of the eye, the white of the eye.

Wherein, step S1 may include: step S11 and step S12,

S11, collecting human body images;

Specifically, equipment such as a camera can be used to collect images of the human body. Preferably, the camera is a monocular camera, and the position and shooting angle of the camera can be adjusted to facilitate the collection of images of the human body.

S12. Recognize the face image in the human body image;

Specifically, step S12 may include: step S121 and step S122,

S121. In the YIQ chromaticity space, according to the distribution interval of the I channel component of the human face skin color, identify the human body skin color image in the human body image, the human body skin color image is framed by a rectangle, and the human body skin color image includes: face image and neck image;

It is understandable that the skin color of the human neck is very similar to the skin color of the human face, while the clothes below the neck are generally different from the skin color. Using a rectangle for frame selection can well frame the face image and neck image.

Those skilled in the art can understand that the YIQ chromaticity space is usually adopted by North American television systems and belongs to the NT SC (National Television Standards Committee) system. Here Y does not refer to yellow, but refers to the color's brightness (Luminance), that is, brightness (Brightness). In fact, Y is the gray value of the image (Gray value), while I and Q refer to the hue (Chrominance), which is the attribute that describes the color and saturation of the image. In the YIQ system, the Y component represents the brightness information of the image, and the I and Q components carry color information. The I component represents the color change from orange to cyan, while the Q component represents the color change from purple to yellow-green.

In the YIQ color space, the skin color of the human face shows better aggregation and compactness on the chroma I channel component. In the YIQ color space, the data analysis and statistics of the I channel component of the image containing the pixel points of the human face skin color can be So, the skin color information is mainly distributed between 40 and 100.

Since the color feature of human lips is also very prominent, in a preferred embodiment of the present invention, the color feature of lips can be further used to determine the human skin color image. Among them, the chromaticity of the lips is distributed within a certain range under the YIQ chromaticity space, and the data of each channel component of the lip color pixel of the image sample containing only the lips is analyzed and statistically found that the maximum value of each channel component is The optimal segmentation thresholds are Y∈[90,210], I∈[20,80], Q∈[10,28] respectively, and the I channel component of skin color and the channel components of lip color information in YIQ chromaticity space The distribution range is used as a criterion for the face region segmentation algorithm.

Of course, step S121 can also be: in the YUV chromaticity space, according to the distribution range of the phase angle θ of the hue distribution of the human face skin color, identify the human body skin color image in the human body image, and the human body skin color image is framed by a rectangle Optionally, the human skin color image includes: a face image and a neck image;

Those skilled in the art can understand that YUV is a color coding method (belonging to PAL) adopted by the European television system, and is a color space adopted by PAL and SECAM analog color television systems. In modern color TV systems, three-tube color cameras or color CCD cameras are usually used to capture images, and then the obtained color image signals are separated, enlarged and corrected to obtain RGB, and then the brightness signal Y and two signals are obtained through a matrix conversion circuit. A color difference signal R-Y (that is, U), B-Y (that is, V), and finally the sending end encodes the three signals of brightness and color difference separately, and sends them out on the same channel. This color representation method is the so-called YUV color space representation. The importance of using the YUV color space is that its brightness signal Y and chrominance signals U and V are separated.

In the YUV chromaticity space, the luminance signal (Y) and the chrominance signal (U, V) are independent of each other, the saturation is determined by the modulus value Ch, the hue is represented by its phase angle θ, and the phase angle of the hue distribution of the human face skin color Theta is distributed between 110 and 155. In the YUV chromaticity space, the phase angle θ of the lip color tone distribution is between 80 and 100, and the distribution range of the phase angle θ of the skin color and lip color in the YUV chromaticity space is used as another criterion for the face region segmentation algorithm. According to it, combined with face region segmentation threshold in YIQ hue space for face detection. Carry out skin color segmentation according to the above threshold value range, and then use morphological opening and closing operations to remove isolated small blocks in the image, and finally use the method of region growth to correct the face area, and mark the face area with a rectangular frame.

S122. Keep the upper short side of the rectangle fixed, shorten the rectangle to become a square, and identify the image inside the square as a human face image.

It can be understood that the skin color area of the lowermost part of the marked rectangular frame after skin color segmentation usually contains the neck of the human body, while the uppermost part of the rectangular frame is the upper edge of the human face area, so keep the uppermost edge of the rectangular frame The position of the rectangular frame remains unchanged, the position of the lower edge of the rectangular frame is reduced, its width remains unchanged, and its height is scaled to be equal to the width.

S13. Identify the eye image in the face image;

Wherein, the step S13 may include:

S131. Divide the face image framed by the square into equal upper half and lower half;

It can be understood that the human eye area is included in the upper half area of the human face area, therefore, the above-mentioned rectangular frame is firstly divided into an upper half area and a lower half area from the middle part, wherein the upper half and the lower half Partial rectangles are equal in length and width.

S132. Divide the upper half into equal left half and right half, wherein the image in the left half carries the image of the right eye region of human eyes, and the image in the right half carries the image of the human eye The image of the left eye region of the eye;

Since the human eye exists in the upper half area, the upper half area is divided from the middle part into a left half area and a right half area, wherein the length and width of the left half area and the right half area are equal, and the human eye The feature information of the left eye of , appears in the right half area.

S133. Use the genetic algorithm to locate the right eye area image of the human eye from the left half of the image, and use the genetic algorithm to locate the left eye area image of the human eye from the right half of the image.

Use the genetic algorithm to find the optimal solution in the right half of the area to locate the left eye area of the human eye, and use the same method to locate the right eye area of the human eye.

S14. Determine the first position of the user's iris center and the second position of the user's eye corner according to the eye image.

Wherein, the method for determining the first position in step S14 may include:

S141. Determine at least three points on the edge of the iris according to the color difference between the iris and the sclera in the binary image;

S142. Determine a first position where the center of the iris is located according to at least three points on the edge of the iris.

Among them, the shape of the human iris can be approximated as a circle, and the geometric properties of the circle can be used. When three points on the circumference of the circle are known, the center of the circle can be determined.

Wherein, the method for determining the second position in step S14 may include:

S143. Estimate the second position by using the first position where the center of the iris is located, and generate an estimated position;

S144. Correct the estimated position by using the corner responsivity function in the improved Harris corner detection algorithm combined with the variance projection function, and determine the second position.

Specifically, the implementation process of step S144 may include: step S144a, step S144b, step S144c, step S144d and step S144e.

S144a. First convert the color image of the human eye area searched by the genetic algorithm into a grayscale image using the conversion formula of a color image and a grayscale image, and then sequentially perform the following processing and positioning on the local images in this small area eye point coordinates;

S144b. Calculate its first-order derivative in the horizontal direction and the vertical direction for each point in the grayscale image of the human eye area, and simultaneously perform a multiplication operation on the image after the derivation in the horizontal direction and the vertical direction, such as formula ( 4.8) as shown:

$\left\{\begin{array}{c}{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; p</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}\\ {\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; p</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}\\ \mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; p</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; x</span>}}<span\; class="notranslate">\; \&CenterDot;</span>\frac{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; p</span>}}{<span\; class="notranslate">\; \&PartialD;</span>\mathrm{<span\; class="notranslate">\; the\; y</span>}}\end{array}\right.$

Formula (4.8)

S144c, use the Gaussian function to perform Gaussian filtering on the image processed in step 2

$\left\{\begin{array}{c}\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; =</span>{{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&CircleTimes;</span>{\mathrm{<span\; class="notranslate">\; exp</span>}}^{<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}\\ \mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; =</span>{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \&CircleTimes;</span>{\mathrm{<span\; class="notranslate">\; exp</span>}}^{<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}\\ \mathrm{<span\; class="notranslate">\; C</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; Z</span>}<span\; class="notranslate">\; \&CircleTimes;</span>{\mathrm{<span\; class="notranslate">\; exp</span>}}^{<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; v</span>}}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; \&CircleTimes;</span>{\mathrm{<span\; class="notranslate">\; exp</span>}}^{<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; 2</span>}{\mathrm{<span\; class="notranslate">\; \&sigma;</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}}\end{array}\right.$

The autocorrelation matrix of the Harris corner detection algorithm can be expressed as:

$\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; =</span>\left[\begin{array}{cc}\mathrm{<span\; class="notranslate">\; A</span>}& \mathrm{<span\; class="notranslate">\; C</span>}\\ \mathrm{<span\; class="notranslate">\; C</span>}& \mathrm{<span\; class="notranslate">\; B</span>}\end{array}\right]$

The responsivity function of the Harris corner detection algorithm is:

C _RF ＝AB-C ² -k(A+B) ²

However, the value of the above formula k needs to be judged based on experience. If the value is improper, the overall detection effect will be reduced. In order to avoid detection errors, another method is introduced instead, as shown in the following formula: where k' is a relatively small integer,

$\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; =</span>\frac{\mathrm{<span\; class="notranslate">\; AB</span>}}{{<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; A</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; B</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; k</span>}}^{<span\; class="notranslate">\; ,</span>}}$

When the replaced responsiveness function is determined, the weighting factor function F of the variance projection function is also determined at this time, and the weighting factor function F is expressed as shown in formula (4.13): where λ is a constant

F＝λ·w

S144d, using the above formula to represent the weighted projection function in the horizontal direction and the vertical direction as shown in the formula:

where I(x, y) is the pixel intensity value of point (x, y).

S144e. Calculate the weighted projection function of the eye corner of the human eye in the horizontal and vertical directions according to the method shown above as the abscissa and ordinate of the eye corner.

S2. Calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance;

S21. According to the first position and the second position, calculate multiple distances between the center of the user's iris and the corners of the user's eyes within a preset time period;

S22. Obtain an average value of the plurality of distances, where the average value is a first distance.

Since the position of the iris is unstable with changes in the external environment, the reliability of the method of mapping the distance relationship between the center of the iris and the corner of the eye point simply by using a certain frame of image positioning is relatively low. The average value of multiple distances between the center of the user's iris and the corners of the user's eyes within a preset period of time is the first distance, which can greatly improve reliability. The actual meaning of this scheme is: the time length of human eyes on a certain area of attention (such as a certain area of the screen) is used as the method for area selection, and within the time period when human eyes are gazing on a certain sub-area, the present invention continuously The collected images are processed, and the distance between the center point of the iris of the human eye and the corner of the eye is mapped to the sub-area of the human eye's gaze on the screen, and the average distance mapped within this time period is used to map the gaze sub-area of the computer screen by the human eye . Wherein, the preset time period can be 5 seconds, because the line of sight of the human eye moves from one gazing sub-region to another gazing sub-region in this time interval, the distance between the center of the iris of the human eye and the corner of the eye and the distance between the sub-regions The mapping relationship of can be stabilized within this period of time, and at the same time, the error caused by the unconscious blinking of the human eye is avoided.

S3. Execute a first operation instruction corresponding to the first distance according to the preset correspondence between the distance and the operation instruction.

Specifically, it is possible to first display pictures on the display screen, walls, blackboards, etc. to guide the user to focus on the displayed part of the content. The displayed content is toilet. When the user looks at the water cup on the left, instructions related to the water cup can be executed, such as telling a nurse that "the patient needs to drink water". When the user looks at the toilet on the right, instructions related to the toilet can be executed, such as telling the nurse that "the patient needs to go to the toilet". Of course, there are many specific implementation manners, which are well known to those skilled in the art, and the present invention will not be repeated here.

Wherein, the first operation instruction may be an operation instruction such as requesting a dressing change, informing of physical discomfort, and requesting a drink of water. It can be understood that the functions implemented by the operation instructions may be various, which is not limited in the present invention, and meanwhile, the corresponding relationship between the distance and the operation instructions may be various, which is also not limited in the present invention.

The method for realizing eye operation provided by the present invention can determine the distance between the center of the user's iris and the corner of the user's eye, and execute the operation instruction corresponding to the distance according to the preset correspondence between the distance and the operation instruction. Since the present invention can realize the eye operation without the user wearing equipment such as a helmet, the user feels good and the cost is low. Since the eye operation can save the user from using the limbs to operate, it is applicable to a wider range of people.

Corresponding to the above method embodiments, the present invention also provides a system for realizing eye operations.

As shown in FIG. 9 , a system for realizing eye operation provided by an embodiment of the present invention includes: a position determination module 100, a distance determination module 200 and an instruction execution module 300,

The position determination module 100 is used to determine the first position where the center of the user's iris is located and the second position of the corner of the user's eye;

Wherein, as shown in Figure 10, the position determination module 100 may include: an image acquisition device 110, a face recognition module 120, a human eye recognition module 130, an iris center recognition module 140 and an eye point recognition module 150,

An image acquisition device 110, configured to acquire an image of a human body;

Specifically, the image acquisition device 110 may be a device such as a camera. Preferably, the camera is a monocular camera, and the position and shooting angle of the camera can be adjusted to facilitate the collection of human body images.

Face recognition module 120, for recognizing the face image in the human body image;

Human eye identification module 130, used to identify eye images in human face images;

The iris center identification module 140 is configured to determine the first position where the user's iris center is located according to the eye image;

The eye point recognition module 150 is configured to determine the second position of the user's eye point according to the eye image.

Wherein, as shown in Figure 11, the face recognition module 120 can include: a first skin color recognition module 121 and a human face frame selection module 122,

The first skin color identification module 121 is used to identify the human body skin color image in the human body image according to the I channel component distribution interval of the human face skin color under the YIQ chromaticity space. The human body skin color image is framed using a rectangle, and the human body skin color image includes: Face image and neck image;

It is understandable that the skin color of the human neck is very similar to the skin color of the human face, while the clothes below the neck are generally different from the skin color. Using a rectangle for frame selection can well frame the face image and neck image.

Those skilled in the art can understand that the YIQ chromaticity space is usually adopted by North American television systems and belongs to the NT SC (National Television Standards Committee) system. Here Y does not refer to yellow, but refers to the color's brightness (Luminance), that is, brightness (Brightness). In fact, Y is the gray value of the image (Gray value), while I and Q refer to the hue (Chrominance), which is the attribute that describes the color and saturation of the image. In the YIQ system, the Y component represents the brightness information of the image, and the I and Q components carry color information. The I component represents the color change from orange to cyan, while the Q component represents the color change from purple to yellow-green.

In the YIQ color space, the skin color of the human face shows better aggregation and compactness on the chroma I channel component. In the YIQ color space, the data analysis and statistics of the I channel component of the image containing the pixel points of the human face skin color can be So, the skin color information is mainly distributed between 40 and 100.

Since the color feature of human lips is also very prominent, in a preferred embodiment of the present invention, the color feature of lips can be further used to determine the human skin color image. Among them, the chromaticity of the lips is distributed within a certain range under the YIQ chromaticity space, and the data of each channel component of the lip color pixel of the image sample containing only the lips is analyzed and statistically found that the maximum value of each channel component is The optimal segmentation thresholds are Y∈[90,210], I∈[20,80], Q∈[10,28] respectively, and the I channel component of skin color and the channel components of lip color information in YIQ chromaticity space The distribution range is used as a criterion for the face region segmentation algorithm.

The human face frame selection module 122 is used to keep the upper short side of the rectangle fixed, shorten the rectangle to make it a square, and identify the image in the square as a human face image.

As shown in Figure 12, in other embodiments of the present invention, the face recognition module 120 may include: a second skin color recognition module 123 and a face frame selection module 122,

The second skin color identification module 123 is used to identify the human body skin color image in the human body image according to the distribution range of the phase angle θ of the color tone distribution of the human face in the YUV chromaticity space. The human body skin color image is framed by a rectangle, and the human body Skin color images include: face images and neck images;

Those skilled in the art can understand that YUV is a color coding method (belonging to PAL) adopted by the European television system, and is a color space adopted by PAL and SECAM analog color television systems. In modern color TV systems, three-tube color cameras or color CCD cameras are usually used to capture images, and then the obtained color image signals are separated, enlarged and corrected to obtain RGB, and then the brightness signal Y and two signals are obtained through a matrix conversion circuit. A color difference signal R-Y (that is, U), B-Y (that is, V), and finally the sending end encodes the three signals of brightness and color difference separately, and sends them out on the same channel. This color representation method is the so-called YUV color space representation. The importance of using the YUV color space is that its brightness signal Y and chrominance signals U and V are separated.

In the YUV chromaticity space, the luminance signal (Y) and the chrominance signal (U, V) are independent of each other, the saturation is determined by the modulus value Ch, the hue is represented by its phase angle θ, and the phase angle of the hue distribution of the human face skin color Theta is distributed between 110 and 155. In the YUV chromaticity space, the phase angle θ of the lip color tone distribution is between 80 and 100, and the distribution range of the phase angle θ of the skin color and lip color in the YUV chromaticity space is used as another criterion for the face region segmentation algorithm. According to it, combined with face region segmentation threshold in YIQ hue space for face detection. Carry out skin color segmentation according to the above threshold value range, and then use morphological opening and closing operations to remove isolated small blocks in the image, and finally use the method of region growth to correct the face area, and mark the face area with a rectangular frame.

The human face frame selection module 122 is used to keep the upper short side of the rectangle fixed, shorten the rectangle to make it a square, and identify the image in the square as a human face image.

It can be understood that the skin color area of the lowermost part of the marked rectangular frame after skin color segmentation usually contains the neck of the human body, while the uppermost part of the rectangular frame is the upper edge of the human face area, so keep the uppermost edge of the rectangular frame The position of the rectangular frame remains unchanged, the position of the lower edge of the rectangular frame is reduced, its width remains unchanged, and the height is scaled to be equal to the width.

As shown in Figure 13, the human eye recognition module 130 may include: a first division module 131, a second division module 132 and a human eye positioning module 133,

The first division module 131 is used to divide the human face image framed by the square into equal upper half and lower half;

It can be understood that the human eye area is included in the upper half area of the human face area, therefore, the above-mentioned rectangular frame is firstly divided into an upper half area and a lower half area from the middle part, wherein the upper half and the lower half Partial rectangles are equal in length and width.

The second dividing module 132 is used to divide the upper half into equal left half and right half, wherein, the image in the left half carries the image of the right eye region of the human eye, and the image in the right half carries the image of the human eye The image of the left eye region of the eye;

Since the human eye exists in the upper half area, the upper half area is divided from the middle part into a left half area and a right half area, wherein the length and width of the left half area and the right half area are equal, and the human eye The feature information of the left eye of , appears in the right half area.

The human eye positioning module 133 is configured to use the genetic algorithm to locate the image of the right eye region of the human eye from the left half of the image, and use the genetic algorithm to locate the image of the left eye region of the human eye from the right half of the image.

Use the genetic algorithm to find the optimal solution in the right half of the area to locate the left eye area of the human eye, and use the same method to locate the right eye area of the human eye.

Wherein, as shown in Figure 14, the iris center identification module 140 may include: an edge point determination module 141 and a center determination module 142,

The edge point determination module 141 is used to determine at least three points on the edge of the iris according to the color difference between the iris and the sclera in the binary image in the eyeball;

The center determination module 142 is configured to determine the first position of the iris center according to at least three points on the edge of the iris.

Among them, the shape of the human iris can be approximated as a circle, and the geometric properties of the circle can be used. When three points on the circumference of the circle are known, the center of the circle can be determined.

Wherein, as shown in FIG. 15 , the corner point identification module 150 may include: an estimation module 151 and a correction module 152,

An estimation module 151, configured to use the first position where the iris center is located to estimate the second position to generate an estimated position;

The correction module 152 is configured to use the corner responsivity function in the improved Harris corner detection algorithm combined with the variance projection function to correct the estimated position and determine the second position.

The distance determination module 200 is used to calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance;

Wherein, as shown in FIG. 16, the distance determining module 200 may include: a distance obtaining module 210 and an averaging module 220,

The distance obtaining module 210 is used to calculate multiple distances between the center of the user's iris and the corner of the user's eyes within a preset time period according to the first position and the second position;

The averaging module 220 is configured to obtain an average value of multiple distances, where the average value is the first distance.

Since the position of the iris is unstable with changes in the external environment, the reliability of the method of mapping the distance relationship between the center of the iris and the corner of the eye point simply by using a certain frame of image positioning is relatively low. The average value of multiple distances between the center of the user's iris and the corners of the user's eyes within a preset period of time is the first distance, which can greatly improve reliability. The actual meaning of this scheme is: the time length of human eyes on a certain area of attention (such as a certain area of the screen) is used as the method for area selection, and within the time period when human eyes are gazing on a certain sub-area, the present invention continuously The collected images are processed, and the distance between the center point of the iris of the human eye and the corner of the eye is mapped to the sub-area of the human eye's gaze on the screen, and the average distance mapped within this time period is used to map the gaze sub-area of the computer screen by the human eye . Wherein, the preset time period can be 5 seconds, because the line of sight of the human eye moves from one gazing sub-region to another gazing sub-region in this time interval, the distance between the center of the iris of the human eye and the corner of the eye and the distance between the sub-regions The mapping relationship of can be stabilized within this period of time, and at the same time, the error caused by the unconscious blinking of the human eye is avoided.

The instruction execution module 300 is configured to execute the first operation instruction corresponding to the first distance according to the preset correspondence between the distance and the operation instruction.

Wherein, the first operation instruction may be an operation instruction such as requesting a dressing change, informing of physical discomfort, and requesting a drink of water.

The system for realizing eye operation provided by the present invention can determine the distance between the center of the user's iris and the corner of the user's eye, and execute the operation instruction corresponding to the distance according to the preset correspondence between the distance and the operation instruction. Since the present invention can realize the eye operation without the user wearing equipment such as a helmet, the user feels good and the cost is low.

For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing the present invention, the functions of each unit can be implemented in one or more pieces of software and/or hardware.

It can be seen from the above description of the implementation manners that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, disk , CD, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments of the present invention.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for related parts, please refer to part of the description of the method embodiment. The system embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

The invention is applicable to numerous general purpose and special purpose computing system environments or configurations. Examples: personal computers, server computers, handheld or portable devices, tablet-type devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, including A distributed computing environment for any of the above systems or devices, etc.

The invention may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them.

The foregoing is only a specific embodiment of the present invention. It should be pointed out that for those of ordinary skill in the art, some improvements and modifications can also be made without departing from the principle of the present invention. It should be regarded as the protection scope of the present invention.

Claims (10)

1. A method for realizing eye operation, comprising: Determine the first position of the user's iris center and the second position of the user's eye corner; Calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance; According to the preset correspondence between the distance and the operation instruction, the first operation instruction corresponding to the first distance is executed. 2. The method according to claim 1, wherein the step of determining the first position of the center of the user's iris and the second position of the corner of the user's eye includes: Collect human body images; identifying a face image in the human body image; identifying eye images in the face image; The first position of the user's iris center and the second position of the user's eye corner are determined according to the eye image. 3. The method according to claim 2, characterized in that, the step of identifying the face image in the human body image comprises: Under the YIQ chromaticity space, according to the distribution interval of the I channel component of the human face skin color, identify the human body skin color image in the human body image, the human body skin color image is framed by a rectangle, and the human body skin color image includes: a human face image and neck image; The upper short side of the rectangle is kept fixed, the rectangle is shortened to become a square, and the image inside the square is recognized as a human face image. 4. The method according to claim 2, characterized in that, the step of identifying the face image in the human body image comprises: In the YUV chromaticity space, according to the distribution range of the phase angle θ of the hue distribution of the human face skin color, identify the human body skin color image in the human body image, the human body skin color image is framed by a rectangle, and the human body skin color image includes : face image and neck image; The upper short side of the rectangle is kept fixed, the rectangle is shortened to become a square, and the image inside the square is recognized as a human face image. 5. The method according to claim 3 or 4, wherein the step of identifying the eye image in the face image comprises: The face image framed by the square is divided into equal upper half and lower half; dividing the upper half into equal left half and right half, wherein the image in the left half carries the image of the right eye region of the human eye, and the image in the right half carries the left eye region image of the human eye. eye area image; The genetic algorithm is used to locate the image of the right eye region of the human eye from the left half of the image, and the genetic algorithm is used to locate the image of the left eye region of the human eye from the image of the right half. 6. The method according to claim 5, wherein in the step of determining the first position of the user's iris center and the second position of the user's eye corner according to the eye image, the first position is determined methods, including: determining at least three points on the edge of the iris according to the color difference between the iris and the sclera in the eyeball in the binary image; The first position where the center of the iris is located is determined according to at least three points on the edge of the iris. 7. The method according to claim 5, wherein in the step of determining the first position of the user's iris center and the second position of the user's eye corner according to the eye image, the second position is determined methods, including: Estimate the second position by using the first position where the center of the iris is located to generate an estimated position; The second position is determined by using the corner responsivity function in the improved Harris corner detection algorithm combined with the variance projection function to correct the estimated position. 8. The method according to claim 1, wherein the distance between the center of the user's iris and the corner of the user's eye is calculated according to the first position and the second position, and the distance is the first Steps for distance, including: According to the first position and the second position, multiple distances between the center of the user's iris and the corners of the user's eyes are calculated within a preset time period; An average value of the plurality of distances is obtained, the average value being a first distance. 9. A system for realizing eye operations, comprising: a position determination module, a distance determination module and an instruction execution module, The position determination module is used to determine the first position where the user's iris center is located and the second position of the user's eye corner; The distance determining module is used to calculate the distance between the center of the user's iris and the corner of the user's eye according to the first position and the second position, and the distance is the first distance; The instruction execution module is configured to execute the first operation instruction corresponding to the first distance according to the preset correspondence between the distance and the operation instruction. 10. The system according to claim 9, wherein the position determining module comprises: an image acquisition device, a face recognition module, an eye recognition module, an iris center recognition module and an eye point recognition module, The image acquisition device is used to acquire human body images; The face recognition module is used to recognize the face image in the human body image; The human eye recognition module is used to recognize the eye image in the human face image; The iris center recognition module is used to determine the first position of the user's iris center according to the eye image; The eye point recognition module is configured to determine the second position of the user's eye point according to the eye image.

Images