JP5062898B2 - User interface device - Google Patents

Description

  The present invention relates to a user interface device that allows a user to specify a function to be executed with respect to a computer device or the like, and more specifically, assigns each function to a fingertip according to the movement of the hand, and each function to the fingertip. It is related with the user interface apparatus which an operator can grasp | ascertain the allocation of this.

  As information equipment terminals become more multifunctional, user interfaces that are easier to use are required. An example of a user interface that is easy to use is a touch panel display. This touch panel display is an intuitive user interface device that can be used to input commands and information by pressing a GUI (Graphical User Interface) displayed on the display, and is intuitive for beginners. However, in the touch panel display, since the touch panel for detecting the contact position of the user's finger and the display are integrated, there is a problem that the operation cannot be executed unless the display is at hand.

  In order to solve this problem, the touch panel and the display are separated, and the user's hand on the touch panel is superimposed on the GUI displayed on the display, thereby maintaining an intuitive operational feeling even from a position away from the display. User interface devices have been developed (see, for example, Patent Document 1).

Patent Document 1 discloses a user interface system that creates a GUI image in which a GUI component is uniquely associated with each coordinate group representing the position of a fingertip. As a result, the functions associated with the respective fingertips are clarified, and what functions can be realized can be presented to the user so as to be visible, and a user interface system that is easier to use can be provided.
International Publication No. 2006/104132 Pamphlet

  However, in the above configuration, since the GUI component and each fingertip are uniquely associated, the functions that can be executed in the same scene are limited to the number of fingers that can be used in that scene. For this reason, it cannot be presented in a visible manner that more functions can be realized.

  The present invention solves the above-described conventional problems, and an object thereof is to provide a user interface device capable of visually presenting that a large number of functions can be realized without being limited to the number of usable fingers. And

In order to achieve the above object, a user interface device of the present invention is a user interface device for operating a device, and includes a hand shape acquisition unit that acquires the hand shape of an operator, and the acquired hand shape. A fingertip position detection unit for detecting a fingertip position; a GUI part data storage unit for storing GUI part data representing a GUI (Graphical User Interface) part uniquely assigned to a function for operating the device; and a hand shape A GUI component allocation unit that allocates GUI component data read from the GUI component data storage unit to each fingertip position of the hand shape acquired by the acquisition unit, and a combination of the GUI component image allocated by the GUI component allocation unit and the hand shape image A superimposed display video creation unit that creates an image and a display unit that displays a composite image created by the superimposed display video creation unit The provided, the longitudinal direction of substantially equal display unit in a direction toward the fingertips of the middle finger from the wrist of the image of the hand shape is displayed on the display unit and the Y-axis direction, X-axis lateral of the display unit that is orthogonal to the Y-axis direction If the direction, the GUI component allocation unit allocates a plurality of GUI components to the at least one finger of the hand shape with a predetermined offset value in the Y-axis direction from the fingertip position, and the superimposed display image creation unit The image of the GUI part follows the fingertip position for the movement of the fingertip position in the X-axis direction, and the image of the hand shape without moving the image of the GUI part for the movement of the fingertip position in the Y-axis direction. A composite image is created.

  According to the user interface device of the present invention, it is possible to visually indicate that a large number of functions can be realized without being limited to the number of fingers that can be used. As a result, a user interface device that is easier to use can be provided.

Hereinafter, a user interface device according to an embodiment of the present invention will be described with reference to the drawings.
(Embodiment 1)
FIG. 1 is a conceptual diagram for explaining a configuration of a user interface device 100 according to the present embodiment. In FIG. 1, the user interface device 100 includes a contact position detection unit 110, a computer 120, and a display 130.

  The contact position detection unit 110 is typically composed of a touch pad that can detect a plurality of contact points. The touch pad detection method includes a capacitance method, a pressure sensitive method, a resistance film method, and the like, but it does not matter which method is selected. Further, the contact position detection unit 110 may be a single camera or a plurality of cameras installed at the upper part or the lower part. Visible light, infrared rays, ultrasonic waves, and the like exist as detection targets of the camera, but it does not matter which detection method is selected. The operator places his / her hand 111 on the contact position detection unit 110 and operates the user interface device 100.

  The computer 120 includes an interface for inputting a signal from the contact position detection unit 110, a CPU or MPU, ROM, RAM, a bus connecting each module, and a video output interface. The computer 120 may be a personal computer (hereinafter referred to as a PC) or a dedicated machine. The computer 120 receives the contact position information 122 acquired by the contact position detection unit 110 as an input and deforms the hand shape model stored in the RAM 121 to create an operator's hand shape image 123. Here, when a camera is used for the contact position detection unit 110, an image acquired by the camera may be used as the hand shape image 123. The RAM 121 stores GUI component data representing a GUI component uniquely assigned to a function for operating the device. The computer 120 reads out GUI component data corresponding to the function assigned to each finger of the hand shape image from the RAM 121, and creates a composite image 131 by superimposing the GUI component image on each finger of the operator's hand shape image 123. And output to the display 130.

  The display 130 is a device that displays the composite image 131 generated by the computer 120. There are liquid crystal, EL, and the like as display methods, but it does not matter which method is selected. The operator performs an input operation by moving the hand 111 on the contact position detection unit 110 while watching the composite image 131 on the display 130.

  Next, when the touch position detection unit 110 is a touch pad that can detect a plurality of touch points, data output from the touch pad will be described with reference to FIG. The touch pad methods for detecting a plurality of points are roughly classified into two methods. In the first method, as shown in FIG. 2A, elements and electrodes that individually detect the presence or absence of contact are arranged on the entire surface, and contact with a finger or the like is determined as a region 201. There are image sensors arranged along with liquid crystal cells. In the second method, as shown in FIG. 2B, elements and electrodes for detecting the presence of a contact object are arranged on a straight line in the vertical and horizontal directions, and a contact area such as a finger is detected in two directions. This is a method of detecting as an intersection 203 of 202.

  Next, an example of an installation method in the case where the contact position detection unit 110 is a single camera installed in the lower part will be described with reference to FIG. FIG. 3A illustrates a cross-sectional view of the contact position detection unit 110 taken along a plane perpendicular to the operation surface. A light emitting unit 302 and a camera 303 are installed below the operation surface 301. The wavelength band associated with the light emitting unit 302 and the camera 303 may be visible light, but may be near infrared or far infrared in order to minimize the influence of external light. When the user places the hand 304 on the top of the operation surface 301, the light emitted from the light emitting unit 302 is reflected by the hand 304 and captured by the camera 303. Thereby, the hand shape image 305 as shown in FIG. 3B can be detected. Note that when the illuminance of light hitting the hand 304 is sufficient to capture an image with the camera, the light emitting unit 302 may be omitted. Further, the light emitting unit 302 and the camera 303 may be configured to image the operation surface 301 from the upper part of the operation surface 301 and the user's hand 304. Also good.

Next, the operation flow of the computer 120 in this embodiment will be described with reference to FIG.
First, the computer 120 inputs contact position coordinates from the contact position detection unit 110 (step S401). Next, mode transition is performed based on the input fingertip coordinates (step S402). Next, based on the user's fingertip coordinates and mode, it is detected whether or not a flick (operation that flicks the finger on the operation surface) or tap is performed (step S403). Next, drawing information is created based on the detection result, and output to the display 130 (step S404). Further, the function desired by the user is executed (step S405). The above operation is repeated every specific time (for example, 10 milliseconds).

Next, a detailed operation flow of step S401 when the contact position detection unit 110 is a touch pad that can detect a plurality of contact points will be described with reference to FIG.
First, the computer 120 inputs contact area data from the contact position detection unit 110. Next, if the area of the contact area is smaller than a predetermined value, it is considered as noise and cut (step S501). The remaining one is recognized as a contact area (step S502). Next, the center of gravity position is determined for each recognized contact area (step S503). Next, the barycenter positions are sorted in the horizontal direction (step S504). For example, the recognized contact areas are named fingertip 1, fingertip 2,... In order from the right of the center of gravity position sorted in the horizontal direction. Next, the sorted barycentric position is stored in the RAM 121 as fingertip coordinates (step S505).

Next, a detailed operation flow in step S401 when the contact position detection unit 110 is a single camera or a plurality of cameras installed at the upper part or the lower part will be described with reference to FIG.
First, the computer 120 inputs imaging data from the camera. Next, binarization processing is performed using a predetermined value as a threshold value. Next, regarding each area of the binarized image, an area having an area smaller than a predetermined value is regarded as noise and cut (step S601). The remaining one is recognized as a hand region (step S602). Next, fingertip coordinates are identified by image recognition for the recognized hand region (step S603). Next, the fingertip positions are sorted in the horizontal direction (step S604). For example, the recognized fingertip positions are named fingertip 1, fingertip 2,... In order from the right of the fingertip positions sorted in the horizontal direction. Next, the sorted fingertip positions are stored in the RAM 121 as fingertip coordinates (step S605).

Next, the detailed operation flow of step S402 will be described with reference to FIG.
First, the computer 120 determines whether or not a trigger for transition to the fingertip button mode is detected based on the input fingertip coordinates (step S701). The selection of the trigger here is arbitrary. For example, it is detected as a trigger that the position corresponding to the button displayed on the screen is tapped or the operation surface is traced with a finger so as to draw a trajectory according to a predetermined rule. Next, when it is determined that a trigger has been detected, the state transitions to the fingertip button mode (step S702). If no trigger is detected, the process of step S402 is terminated without doing anything.

Next, the detailed operation flow of step S404 will be described with reference to FIG.
First, when the computer 120 transits to the fingertip button mode, the information of the GUI component is read together with the offset information (step S801). The configuration of the GUI component information and offset information will be described later. Next, fingertip coordinates corresponding to the GUI component are referred to (step S802). Next, for each fingertip coordinate, a position that takes into account the offset information of the GUI part read in step S801 is calculated, and a command group for drawing the GUI part at that position is generated (step S803). Here, the actual drawing is not performed only by setting the command in the drawing queue. Finally, it is checked whether or not the necessary GUI component drawing processing has been completed (step S804). If not completed, the process returns to step S801 to draw the remaining GUI parts. When the drawing process in which the necessary GUI component is superimposed on the hand shape image is completed, the process of step S404 is terminated.

Next, an example of the format of the GUI component data with offset information read in step S801 will be described with reference to FIG.
The first row of the table in FIG. 9 is the GUI component ID. A unique value is assigned to each part. The second line is a number or character string that represents the type of GUI component, and is assigned a value that represents the type of display form, such as a command button, toggle switch, or check box. The third line is shape data, and a value representing a size such as a rectangle and a shape such as a polygon is assigned. The fourth line is color and texture data, which is bitmap data pasted on the surface of the GUI component and data related to colors used for painting and frame lines. The fifth line is text data, and a character string described as a label is assigned to the surface of a command button, toggle switch, or the like. The sixth line is an execution command, and an ID of a command to be executed when a GUI component is selected is assigned. The seventh line is a corresponding finger, and the identification (thumb, index finger, etc.) of the finger to be followed by the GUI component is stored. The eighth line is a vertical direction (Y-axis direction) offset from the fingertip coordinates, and is an offset value from the fingertip coordinates at the time of mode transition to the position (typically the center position) at which the GUI component is to be displayed.

  For example, it is assumed that the corresponding finger is a middle finger and the offset value is +10. When the coordinate of the middle finger at the time of the mode transition is (x, y) = (30, 80) in the screen coordinate system, the position where the GUI component is to be displayed at the time of the mode transition is set to the offset value in the vertical y coordinate. In addition, (30, 90) is obtained. In this case, the GUI component corresponding to the upper part of the fingertip position is drawn.

  Next, a process when the process of creating a composite image by superimposing a GUI part on a hand shape image is completed will be described with reference to the flowchart of FIG. . FIG. 10 shows the second and subsequent processing, which is performed in place of the flowchart of step S404 in FIG. Note that the same processes as those in FIG. 8 are denoted by the same step numbers and description thereof is omitted. The difference from FIG. 8 is that the processes of steps S1001 and S1002 are added. In the first drawing process of FIG. 8, a composite image in which a GUI part is superimposed on the hand shape image is already displayed on the display. Here, the computer 120 compares the latest Y-coordinate of the fingertip coordinates stored in the RAM 121 with the previous Y-coordinate to determine whether or not it has changed by a predetermined value or more, and determines that it has changed by a predetermined value or more. (Yes in step S1001), the process proceeds to step S1002, and the fingertip coordinates are set to the previous fingertip coordinates.

  That is, when the user moves the hand in the X-axis direction, the processing of steps S801 to S804 is performed using the latest fingertip coordinates, and the GUI component moves following the movement of the fingertip of the hand shape image. When the Y-coordinate of the fingertip coordinate changes as the user bends or stretches the finger, the drawing process in steps S801 to S804 is executed with the previous fingertip coordinate, so the GUI component follows the movement of the fingertip. Does not move.

Next, an example of the screen drawn in step S404 will be described with reference to FIGS.
FIG. 11 is an example in which a cross cursor key is displayed at the time of fingertip button mode transition. In FIG. 11, a left key 1105, a determination key 1106, and a right key 1107 are drawn at positions corresponding to the fingertip coordinates 1102 of the ring finger, the fingertip coordinates 1103 of the middle finger, and the fingertip coordinates 1104 of the index finger. The offset values in FIG. 9 are not set for these buttons. On the other hand, the up key 1108 is drawn above the fingertip coordinates 1103 of the middle finger. This is because an offset value is set for the GUI component corresponding to the up key 1108. Similarly, the lower key 1109 is drawn below the fingertip coordinates 1103 of the middle finger according to the offset value. As a result, three functions are assigned to the middle finger.

  If the user wants to select the upper key 1108, the upper key 1108 can be selected by extending the middle finger and tapping upward from the position at the time of mode transition. When the finger is extended, the fingertip coordinates are shifted upward (Y-axis positive direction), but the position of the assigned GUI component does not move with respect to the movement of the finger in the Y-axis direction. Therefore, the upper key 1108 can be selected by extending the middle finger and tapping upward.

When the user wants to select the lower key 1109, the lower key 1109 can be selected by bending the middle finger and tapping downward from the position at the time of mode transition. When the finger is bent, the fingertip coordinates are shifted downward (Y-axis positive direction), but the position of the assigned GUI component does not move with respect to the movement of the finger in the Y-axis direction as described above. Therefore, the lower key 1109 can be selected by bending the middle finger and tapping downward.

  The Y-axis direction is substantially equal to the direction from the wrist toward the fingertip of the middle finger, and a desired GUI component can be easily selected as described above by bending or stretching the finger.

  In this way, by assigning a GUI part representing the cross cursor key to each fingertip, and particularly by assigning three GUI parts to the middle finger, each direction indicated by the cross cursor key matches the position of the button. These keys can be selected by a typical operation.

  FIG. 12 is an example of a display screen when the hand moves to the right from the fingertip position at the time of mode transition. Since each GUI component is set with an offset value in the vertical direction (Y-axis direction) from the fingertip, not in the absolute coordinates to be displayed, each GUI component follows the movement of the hand in the horizontal direction (X-axis direction). And move. As a result, even if the user's hand moves unintentionally from the time of mode transition (for example, when the hand shakes in a vibrating environment), the button will follow and the button will continue to operate. It can be performed.

Next, an example of a screen displayed in the present embodiment will be described with reference to FIG.
FIG. 13 shows an example in which a plurality of numeric keys mounted on a mobile phone or the like are assigned to each finger. In this way, by assigning a plurality of buttons with offset values to each finger, it is possible to execute functions that are more than the number of fingers. Since the assignment of the function of each finger is also divided by bending or extending the finger, it is easy to push the finger, and a blind operation is also possible when used.

Next, an example of a place where the user interface device is installed in the vehicle will be described with reference to FIGS.
FIG. 14 shows a case where it is installed on the center console. By installing the touch pad 1402 on the center console with respect to the display 1401, it is possible to perform an operation at hand while looking at a distant screen. FIG. 15 shows a case where the touch pad 1502 is installed on the handle. By installing the touch pad 1502 in the center portion of the handle 1503, it is possible to perform an operation at hand while looking at a distant screen as in FIG. In any case, by performing the display as shown in FIG. 13, a large number of functions can be activated by a blind operation.

  According to the user interface device of the present invention, it is possible to visually indicate that a large number of functions can be realized without being limited to the number of fingers that can be used. It can be used for input devices.

The conceptual diagram for demonstrating the structure of the user interface apparatus concerning embodiment of this invention. It is a figure for demonstrating the detection system of the touchpad which can detect the some contact point used as the contact position detection part 110, (a) It demonstrates the system by which the element and electrode which detect the presence or absence of a contact are arrange | positioned in the whole surface (B) The figure for demonstrating the system which detects a contact area as an intersection of the detection result of the vertical and horizontal direction It is a figure which shows an example of the installation method in case the contact position detection part 110 is a single camera installed in the lower part, (a) Sectional drawing when the contact position detection part 110 is cut by the surface perpendicular | vertical to the operation surface, (B) The figure which shows the hand shape image imaged with the camera The flowchart which shows operation | movement of the computer 120 of the user interface apparatus in this Embodiment. The flowchart which shows detailed operation | movement of step S401 of FIG. 4 in case the contact position detection part 110 is a touchpad which can detect a some contact point. A flowchart showing the detailed operation of step S401 in FIG. 4 in the case where the contact position detection unit 110 is a single camera or a plurality of cameras installed at the top or bottom. The flowchart which shows the detailed operation | movement of step S402 of FIG. The flowchart which shows the detailed operation | movement of step S404 of FIG. The figure which shows an example of the format of the GUI component data with offset information read by step S801 of FIG. Flowchart showing the detailed operation of step S404 processed in place of the flowchart of FIG. 8 after the second time. The figure which shows an example of the screen drawn by step S404 of FIG. The figure which shows an example of the screen drawn by step S404 of FIG. The figure which shows an example of the screen drawn by step S404 of FIG. The figure which shows an example in which the touchpad of the user interface device concerning embodiment of this invention was installed in the center console in a vehicle The figure which shows an example in which the touchpad of the user interface apparatus concerning embodiment of this invention is installed in the handle | steering_wheel in a vehicle

Explanation of symbols

100 User interface device 110 Contact position detection unit 111, 304 Hand 120 Computer 121 RAM
122 Contact position information 123, 305, 1101, 1211 Hand shape image 130, 1401, 1501 Display

Claims (1)

A user interface device for operating a device,
A hand shape acquisition unit for acquiring the hand shape of the operator;
A fingertip position detector that detects the fingertip position of the acquired hand shape;
A GUI part data storage unit for storing GUI part data representing a GUI (Graphical User Interface) part uniquely assigned to the function for operating the device;
A GUI component allocation unit that allocates GUI component data read from the GUI component data storage unit to each fingertip position of the hand shape acquired by the hand shape acquisition unit;
A superimposed display video creation unit that creates a composite image of a GUI part image and a hand-shaped image assigned by the GUI part assignment unit;
A display unit for displaying the composite image created by the superimposed display video creation unit,
The vertical direction of the display unit, which is substantially equal to the direction from the wrist of the hand-shaped image displayed on the display unit toward the fingertip of the middle finger, is the Y-axis direction, and the horizontal direction of the display unit orthogonal to the Y-axis direction is X Axis direction
The GUI component allocating unit allocates a plurality of GUI components to at least one finger of the hand shape with a predetermined offset value in the Y-axis direction from the fingertip position,
The superimposed display video creation unit causes the image of the GUI component to follow each fingertip position with respect to the movement of the fingertip position in the X-axis direction, and the GUI component with respect to the movement of the fingertip position in the Y-axis direction. A user interface device, wherein a composite image with a hand-shaped image is created without moving the image.

Images