JP2013114647A - Gesture input system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a gesture system that has a character input function applicable to practical use and is capable of recognizing a movement of a hand part below the elbow with less physical burden on a user and responding to a quick gesture.SOLUTION: An image sensor 3 images a movement of mainly a hand 2 of a user 1. The gesture system performs image processing and recognition in real time on the captured video image, and displays the result on a screen 4 of a television or a smart phone, so that the user 1 can easily operate a device such as a television with a gesture of the hand 2 without burden.

Description

  The present invention relates to a user interface technology for operating devices such as a television, a game, and a smartphone by a gesture.

  In the conventional gesture system, the resolution for recognition is not so high, and it is difficult to recognize the movements of hands and fingers. For this reason, as shown in Patent Document 1, a large movement, that is, the movement of the entire hand (arm) or the entire body (body action) is basically used. Therefore, 1) The physical burden on the user is large. 2) It is difficult to operate a small object (machine control icon) on the screen. There were problems such as. In addition, the frame rate is low (30 to 60 FPS), which makes it difficult to respond to a user's quick gestures, which also deteriorates usability.

  The remote control system is equipped with many buttons to execute commands by operating the buttons on the remote control, but it takes time to fully understand each of these functions and operate them accurately. . There is also a problem that it cannot be used when it is lost or with dirty hands. Character input is even more difficult, and some remote controllers have a small keyboard, but in order to enter characters smoothly while looking at a distant monitor, blind touch is required.

  Many smartphone user interfaces are based on touch panels. The problem with the touch-panel-based user interface is that when you touch the panel with your finger etc., the object such as buttons and keys drawn on the panel is covered with your finger etc., so the user cannot accurately recognize the position of the object There is. This is often a problem for adults with large fingers or Westerners (so-called Fat Finger Problem).

Japanese Patent Laid-Open No. 10-31551

  The problems to be solved are as described above, and the present invention is capable of recognizing the movement of the hand part ahead of the elbow with less physical burden on the user, has a character input function that can withstand practical use, and is a quick gesture. It was made for the purpose of providing a gesture system that can handle the above.

  For this reason, the present invention provides a shooting means for shooting a motion of an operator's hand and fingers, an operation panel display means for displaying an operation panel on the screen of the operation target device, and a pointer moving in accordance with the movement of the shot hand. A pointer display means for displaying, and capturing the motion of the operator's hand and fingers with a high-speed and high-resolution image sensor, performing image processing and recognition on the captured image in real time, By displaying the operation panel on the screen of the operation target device, the most important feature is that the operator can easily operate the operation target device without a load by gesture operations of hands and fingers.

  The present invention captures the motion of the operator's hand and fingers with a high-speed, high-resolution image sensor, so that even a small object on the screen can be accurately and quickly operated in response to a quick motion of the hand or finger. Provide a comfortable feeling to use. By responding to a quick action, another gesture, for example, a moving object in a moving image can be selected and a command can be executed on that object. In addition, the user is not burdened by the gesture operation of the hands and fingers mainly above the elbow. Character input can be done quickly and comfortably with both hands.

It is a block diagram of the gesture input system which implemented this invention. It is a flowchart of a high-speed image calculation process.

  Embodiments of the present invention will be described below.

FIG. 1 shows a configuration diagram of a gesture input system embodying the present invention.
The gesture input system captures mainly the motion of the hand 2 of the user 1 with the image sensor 3, performs image processing and recognition on the captured image in real time, and outputs the result to the screen 4 of a TV, a smartphone or the like. The display realizes that the user 1 can easily operate a device such as a television by a gesture operation of the hand 2 without load. The image sensor 3 has a high-speed image arithmetic processor integrated therein, and is compactly mounted on a housing such as a television. The gesture can be an action above the wrist (including a fingertip and a single finger), and the high-speed image processor recognizes the shape of the hand, the finger, etc. of the gesture action.

  In this gesture system, the image sensor 3 having a high speed of 100 FPS or higher and a high resolution of 640 × 480 or higher, which has not been used in this field in the past, can be used to accurately capture the quick movement of the human hand 2 and fingers during the gesture. Adopting the hand 2 and finger gesture motions quickly and accurately, and providing a comfortable user interface unprecedented to TVs, game machines and the like.

  In order to realize a user interface comfortable for the user 1, not only an operation with one hand, which is seen in a conventional system, but also a gesture using both hands is provided.

  In this gesture system, 1-2 pointers (hand icons) are displayed on the screen, and simple gesture operations using the fingers of both hands are used to search for various operations, for example, programs to be viewed. And selection (equivalent to switching TV channels, selecting programs to watch from the program guide, etc.), displaying and moving multiple windows on the screen, changing the size of the windows using both hands or one hand, displayed on the screen It is designed so that characters can be input and commands can be specified by operating two pointers on the virtual keyboard with a finger.

  Since the user 1 realizes the gesture only by operating the two fingers ahead of the elbows of both arms, the user 1 can perform the operation continuously without fatigue even with the daily operations such as television.

  When a gesture operation using all or part of the fingers of both hands 2 with a large screen 5 such as a TV is suitable, the keyboard used for character input corresponds to the natural movement of the hands 2 With a key layout and design.

  In addition to the position and movement information of hand 2 operation, in order to realize gesture tap operation (operation that realizes the situation where a key on the keyboard or a button on the menu is pressed) at high speed and stably, The speed and acceleration are used for recognition.

  Recognition of the hand 2 and the finger is performed using a plurality of appropriate feature amounts related to the finger obtained from the image sensor.

  In the case of a smartphone, tactile feedback is performed by the vibration element on the back of the device.

  Zooming is realized by the back and forth movement of the finger and hand 2 with the target device so that 3D video can be drawn.

FIG. 2 shows a flowchart of high-speed image calculation processing.
In the high-speed image calculation process, first, environmental parameters of the sensor are set (parameters are adjusted as needed according to environmental changes) (step 1), then an image is captured (step 2), and then gesture operation is performed. The position, shape, and the like of the hand, the finger, etc. are comprehensively judged and identified over a plurality of frames (step 3). Next, extraction and calculation of identified hand / finger attributes / features (characteristic hand, finger part position, size, shape, motion speed, etc.) were performed (step 4), and then identified. The gesture operation is recognized from the hand attributes / features and provided to the system (OS) (step 5). Next, the system program and application program obtain the gesture operation information from the system (OS), and the gesture UI. (User interface) is realized (step 6).

  In other words, the gesture subsystem captures the user's gesture motion as a moving image, and through image processing and gesture recognition processing there, the hand, finger shape, relative movement amount, speed acceleration, etc. are measured, and the gesture is determined from those data. The process of recognizing the meaning of the operation is performed, and then the position of the pointer on the image, the amount of movement, and the state at that point (data indicating the state and its meaning, state change, etc.) are created, and the TV set is used as a data set. Provide a side system to realize a user interface system by gesture.

  With the above configuration, when the user shakes his / her hand arbitrarily and quickly, the gesture system recognizes the hand and its movement in the image and starts gesture recognition. At this time, calibration is performed to match the arm search range indicated by the dotted line in FIG. 1 with the screen screen so that all the points on the screen can be operated by moving the hand portion beyond the elbow.

  Character input is realized by the operation of fingers of both hands, a virtual keyboard optimized for gesture character input is displayed on the screen, and the keys drawn there are operated by operating two pointers with fingers of both hands To do. The gesture of selecting and pressing a key is realized by quickly and accurately recognizing the posture, movement, speed, and acceleration of the finger.

  Further, it measures and recognizes the hand, finger posture, relative movement, speed, and acceleration relative to the movement of the entire hand of the finger to recognize taps, drags, flicks, and the like.

  This gesture system provides, for example, a 3D video rendering function on a screen using a 3D position of a finger in a smartphone. In addition, a 3D position of the finger and in this case the eye is defined as the viewpoint, and a 3D image is displayed on the screen. To do. For example, this provides a function for controlling a street view such as a map.

  Also, if the gesture system operating environment (living room, etc.) is in an environmental state where malfunction may occur, the cause of the malfunction (hereinafter referred to as an obstacle. For example, the presence of a very strong light source, In order to suggest to the user what is inappropriate illuminance, etc., remove it, and solve the problem, display the video information in one window on the screen and recognize the presence of the obstacle to the user Provide a function that allows you to remove and remove.

  In addition, hand, finger position, movement, acceleration, and shape are recognized by calculating contour data used from hand color information, color gradient information, etc. By comparing, the hand, finger position, movement, acceleration, and shape are recognized.

1 user 2 hand 3 image sensor 4 screen

Claims (6)

Photographing means for photographing the movement of the operator's hand and fingers;
Operation panel display means for displaying the operation panel on the screen of the operation target device;
Pointer display means for displaying a pointer as a movie in conjunction with the movement of the photographed hand;
With
The operator's hand and finger movements are imaged with a high-speed, high-resolution image sensor, and the captured images are processed and recognized in real time. The results and the operation panel are displayed on the screen of the operation target device. A gesture input system characterized in that, by displaying, an operator can easily operate an operation target device without a load by a gesture operation of a hand and a finger.   The gesture input system according to claim 1, wherein the pointer has a shape that appropriately associates movements of an operator's hand and fingers.   The gesture input system according to claim 1, wherein the gesture operation includes a two-hand operation.   The screen control includes the basic operations such as TV channel switching, volume adjustment, program search and program selection, movement of a plurality of windows displayed on the TV screen, resizing, and still images included in videos for online shopping on the TV. Or, selection of moving items, input of characters used for communication via television such as social network applications, input of characters for searching programs, and operations for voting to collect viewers' opinions The gesture input system according to claim 1, further comprising:   The gesture input system according to claim 1, wherein an operation panel to be displayed on the screen is selected by the gesture operation. The gesture operation is a flick operation that scrolls the screen or scroll bar with inertia,
Drag operation to move the display window on the screen,
Pinch operation to enlarge / reduce the screen display,
Tap operation to press keys on the screen and enter keys,
Slide operation to move the lever on the screen,
Rotation operation to rotate the volume on the screen,
Search operation to search the menu on the screen,
The gesture according to claim 1, comprising any one of operations with both hands and fingers of a keyboard on the screen for inputting an operation character for selecting a stationary and moving object displayed on the screen. Input system.