JP5401675B1 - Information input device and information input method - Google Patents

Abstract

An object of the present invention is to make it possible to correctly recognize the movement of an operator's hand by narrowing down the range of an operation target screen and to appropriately perform an operation on an object performed based on the movement of the hand.
An information input device includes a display unit, an area setting unit for setting an input instruction area for an operator to input an input, and an acquisition unit for acquiring a situation of the operator who performs the input instruction. In accordance with the movement of both hands of the operator determined based on the acquired situation information, the display area is associated with a part of the entire display area, and the operator's The control unit 105 includes a selection area for accepting a selection operation and a determination area for accepting an operator's determination operation, which are separately arranged on the input instruction area.
[Selection] Figure 7

Description

  The present invention relates to an information input device and an information input method, and more particularly to an information input device and an information input method for performing an input operation using the movement of an operator's hand.

  2. Description of the Related Art Conventionally, there is known an information input device that gives an input instruction to a computer by moving an operator's hand and performs an input operation performed based on the input instruction. For example, Patent Document 1 includes a CCD camera, a computer for recognizing the shape of an object in an image captured by the CCD camera, and a display for displaying the object recognized by the computer. An apparatus for performing a selection operation of a cursor displayed on a display is disclosed.

  Patent Document 2 discloses an apparatus that performs, for example, a drag operation by the movement of an operator's hand on a virtual plane.

JP 2004-079777 A JP 2004-258714 A

  In the apparatuses disclosed in Patent Documents 1 and 2, the range of the target screen that the operator moves and operates is wide. However, in a situation where the range of the target screen to be operated is wide, the movement of the operator's hand is not correctly recognized, and the conventional apparatus moves the object such as the cursor to be selected along with the movement of the hand. There was a problem that could not be performed properly.

  Accordingly, the present invention provides an information input device that can correctly recognize the movement of the operator's hand by narrowing the range of the operation target screen and appropriately perform an operation on the object performed based on the movement of the hand. An object is to provide an information input method.

An information input device for achieving such an object includes a display section having a display area, an area setting section for setting an input instruction area for an operator to input instructions, and the input instruction. Corresponding to a part of the entire display area according to the action of both hands of the operator, which is determined based on the acquired situation information, and an acquisition unit for acquiring the operator's situation And a control area for distinguishing and arranging a selection area for accepting the operator's selection operation in the partial area and a decision area for accepting the operator's decision operation on the input instruction area. The control unit arranges the selection area and the determination area in accordance with a pattern of the finger of the operator recognized from a change in the operation of both hands of the operator .

In addition, an information input method for achieving the above object includes a step of acquiring a situation in which an operator gives an input instruction, and an operation of both hands of the operator determined based on the acquired situation information. A selection area for accepting the operator's selection operation in the partial area, and a selection area for accepting the operator's decision operation. A step of distinguishing the determined region from the determined region, and the step of arranging the selected region and the determined region according to a pattern of the operator's finger recognized from a change in operation of both hands of the operator. And arrange them .

  Further, a program for achieving the above object is for causing a computer to execute the above information input method.

  According to the present invention, it is possible to correctly recognize the movement of the operator's hand by narrowing down the range of the operation target screen, and appropriately perform an operation on the object performed based on the movement of the hand.

It is a figure which shows an example of schematic structure of the information input device which concerns on embodiment of this invention. In an information input device of an embodiment, it is a figure showing an example of a mode in which an operator gives an input instruction. It is a figure which shows an example of the positional relationship of each area | region when seeing the selection area | region and determination area | region set when an operator performs input instruction | indication from the top. It is a figure which shows an example of the positional relationship of each area | region when the selection area | region and determination area | region set when an operator performs input instruction are seen from the side. It is a figure which shows an example of the change aspect of a hand movement in case a selection area | region and a determination area | region interchange. It is a figure which shows an example of an operator's instruction | indication aspect in case the pointer displayed on a display apparatus moves according to the motion of the finger in a selection area. It is a figure which shows an example of a function structure of the information input device which concerns on embodiment. It is a flowchart which shows the operation example of the information input device which concerns on embodiment. It is a figure which shows the modification in case the number of cameras provided in an information input device is four. It is a figure which shows the modification which made the selection area | region and the determination area | region set on a desk surface.

  Hereinafter, an information input device according to an embodiment of the present invention will be described. The information input device according to the embodiment causes the operator to perform an input operation according to the movement of the operator's hand.

[Configuration of information input device 1]
FIG. 1 is a diagram showing a hardware configuration example of an information input device 1 according to an embodiment of the present invention. As shown in FIG. 1, an information input device 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a camera 14, a display device 15, and an input. Device 16.

  The CPU 11 is connected to each component via a bus to perform control signal and data transfer processing, and also executes various programs and arithmetic processing for realizing the operation of the entire information input device 1.

  The ROM 12 stores programs and data necessary for the operation of the entire information input device 1. This program is stored in a recording medium such as a DVD-ROM, and is read out to the RAM 13 to be executed by the CPU 11 to realize the information input device 1 of the present embodiment.

  The RAM 13 temporarily stores data and programs.

  The camera 14 captures the situation of the input instruction made by the operator, and the image is transmitted to the CPU 11 for image recognition. As will be described later, in the information input device 1 of this embodiment, the camera 14 captures a two-dimensional or three-dimensional image for recognizing an input instruction given by an operator. Any system can be used as the configuration of the camera 14. The camera 14 is, for example, a camera with a CCD (Charge Coupled Device) sensor, a camera with a CMOS (Complementary Metal Oxide Semiconductor) sensor, an infrared camera, or the like.

  The display device 15 can be a flat panel display such as a liquid crystal display or EL (Electro-Luminescence).

  The input device 16 includes, for example, a keyboard, a mouse, operation buttons, a touch panel, an input pen, a sensor, and the like.

[Outline of input instructions]
Next, an outline of an input instruction by the operator realized by the information input device 1 will be described.

  First, an aspect of an operator's input instruction will be described with reference to FIGS. FIG. 2 is a diagram illustrating an example of an input instruction form of the operator. FIG. 3 is a diagram illustrating an example of the positional relationship of each region when the selection region and the determination region are viewed from above. FIG. 4 is a diagram illustrating an example of the positional relationship of each region when the selection region and the determination region are viewed from the side.

  As shown in FIG. 2, in the information input device 1, a camera 14 is attached to the display device 15, and the camera 14 performs a situation where the operator gives an input instruction, that is, the movement of both hands 502 and 503 of the operator 500. It is configured to take an image.

  The information input device 1 recognizes the finger shape of both hands 502 and 503 of the operator from the image captured by the camera 14, and, as a result, selects the selection area R1 and the determination area R2 as virtual input instruction areas. It is configured to be arranged separately.

  The selection operation is a selection instruction for an object (icon, pointer, cursor, etc.) displayed on the display device 15. The decision operation is a decision instruction such as a click. The selection operation and the determination operation will be described in detail later.

  For example, in the example of FIG. 2, when the operator 500 opens the left hand 502 and raises the index finger of the right hand 503, the CPU 11 selects the operator 500 on the right hand 503 side according to the shape of the finger of the operator 500. A selection area R1 for accepting an operation is arranged, and a decision area R2 for accepting the decision operation of the operator 500 is arranged on the left hand 502 side.

  In the following description of the embodiment, a hand-shaped pattern with the index finger raised is referred to as a “hand pattern” of the selection region R1. In the information input device 1 of this embodiment, in the hand pattern accompanying the input instruction of the operator 500, the selection region R1 is arranged on the hand side. Further, the determination region R2 is arranged on the other hand side. This means that each time the CPU 11 recognizes the hand pattern of the selection area R1, the selection area R1 and the determination area R2 are interchanged and arranged so that the selection operation by the operator 500 is performed. For example, it is performed with the index finger raised.

  The two selection areas R1 and determination area R2 shown in FIG. 2 are arranged in a predetermined virtual input operation area. For example, as shown in FIGS. 2 to 4, both the selection area R <b> 1 and the determination area R <b> 2 are ranges that connect the operator's viewpoint 501 and the four corners of the display area of the display screen 15 (dotted lines in FIGS. 2 to 4). It is provided not to enter. As shown in FIG. 3, a selection area R <b> 1 and a determination area R <b> 2 are arranged between the keyboard as the input device 16 and the display device 15. Thereby, the camera 14 can image so that the movement of both hands of the operator 500 can be recognized.

  The positions where the selection region R1 and the determination region R2 are arranged are not limited to the above-described example, and can be changed as long as the situation of the input instruction by the operator 500 can be recognized.

[Replacement of areas R1 and R2]
Next, the movement of the hand of the operator 500 for arranging the selection area R1 and the determination area R2 in a switched manner will be described with reference to FIG. FIG. 5 is a diagram illustrating an example of a change in movement of the hands 502 and 503 when the selection area R1 and the determination area R2 are arranged interchangeably, and (a) a state of the selection operation by the right hand 503, (B) The state of selection operation by the left hand 502 is shown.

  In FIGS. 5A and 5B, the selection region R1 is arranged on the hand side by raising the index finger of either hand. In this example, the right hand 503 changes from a state where the index finger is raised → a state where all fingers are opened, and the left hand 502 changes from a state where all fingers are opened → a state where the index finger is raised. In the information input device 1, along with this change, the selection region R1 is arranged by switching from the right hand 503 to the left hand 502 side, and the decision region R2 is arranged by switching from the left hand 502 to the right hand 503 side.

[Operator's selection operation in selection area R1]
Next, the selection operation of the operator 500 within the selection region R1 will be described with reference to FIG. FIG. 6 is a diagram showing an example of the operation mode of the operator 500 when the pointer 40 displayed on the display device 15 moves in response to the movement of the finger in the selection region R1, and (a) the right hand 503 is shown. And (b) a selection operation with the left hand 503 is shown.

  In FIGS. 6A and 6B, the selection area R1 is associated with a partial area 151 (shown by hatching in the drawing) of the entire display area of the display device 15. In this information input device 1, the above-described area 151 is set to be larger than the area equally divided into two at the center of the display area (indicated by a one-dot chain line in the vertical direction in the figure).

  In the example of FIG. 6A, since the selection area R1 is arranged on the right hand 503 side, a part of the area 151 associated with the selection area R1 is, for example, one point in the vertical direction in the display area. It is set including the region on the left side of the chain line. Therefore, a display area in which the operator 500 can perform a selection operation in the selection area R1 is an area 151 illustrated in FIG.

  In FIG. 6A, when the index finger of the right hand 502 is moved in the selection area R1, the pointer 40 on the area 151 associated with the position of the index finger 20 moves along with the movement. It is indicated by a solid line indicated by reference numerals 20 and 30. In this way, a selection operation within the selection region R1 becomes possible.

  On the other hand, a part of the area 151 associated with the selection area R1 shown in FIG. 6B is set to include the area on the right side of the center of the display area (indicated by a dashed line in the vertical direction in the figure). The

  The determination of the selection operation in the selection area R1 shown in FIGS. 6A and 6B is performed, for example, so that the selection area R1 associated with the area 151 shown in FIGS. 6A and 6B is imaged. The angle of view of the camera 14 is set, and this is performed based on the image from the camera 14 having the above-described angle of view. The association between the selected region R1 and the region 151 is performed, for example, according to a mapping table for converting the coordinate data of each region. For example, as shown in FIG. 6A, when the operator 500 moves the index finger in the selection region R1, the CPU 11 stores the region 151 associated with the position (coordinate data) on the selection region R1 pointed to by the index finger. The “pointer 40 selection operation” is determined. Then, the CPU 11 moves the pointer 40 in the area 151 as the index finger moves in the selection area R1.

  The range of the area 151 associated with the selection area R1 is not limited to the above-described example, and can be changed by setting so that the entire display area is divided into two areas <area 151 <total display area.

  As described above, in the information input device 1 according to the present embodiment, the selection area R1 and the determination area R2 are switched and arranged according to the movement of the hand of the operator 500. Will be used to make an appropriate selection operation. This means that when the operator 500 gives a selection instruction with a hand near the object (icon or the like) to be selected, the object is selected and the selection operation is performed more intuitively. This improves operability.

[Operator's decision operation in decision area R2]
Next, the determination operation of the operator 500 within the determination region R2 will be described with reference to FIG.

  In the information input device 1, when the operator 500 performs a predetermined operation using the left hand 502 in the determination region R2 shown in FIG. As an example of the predetermined operation, when the left hand 502 is pushed out from the determination region R2 toward the display device 15, the left hand 502 in the determination region R2 changes from an open state to a closed state to an open state. is there.

[Functional configuration of information input device 1]
FIG. 7 is a diagram illustrating an example of a functional configuration of the information input device 1 realized on the hardware configuration illustrated in FIG. 1.

  As illustrated in FIG. 7, the information input device 1 includes a display unit 101, an acquisition unit 102, a storage unit 103, an area setting unit 104, and a control unit 105.

  The display unit 101 is configured by the display device 15 in FIG. 1 and is provided to make an object as a selection target visible to an operator.

  The acquisition unit 102 acquires a situation when the operator gives an input instruction. In this embodiment, the acquisition unit 102 is configured by the camera (imaging unit) 14 in FIG. 1, for example, and is provided for imaging the situation when the operator issues an input instruction. Note that the acquisition unit 102 may apply another configuration known to those skilled in the art, for example, a motion sensor that senses the movement of the operator's hand, as long as the situation when the operator gives an input instruction can be acquired. You can also.

  The storage unit 103 is configured by the ROM 12 and the RAM 13 in FIG. 1 and stores data.

  The region setting unit 104 and the control unit 105 function by the CPU 11. The area setting unit 104 sets an input instruction area for the operator 500 to input an instruction. The input instruction area is a virtual area for arranging the selection area R1 and the determination area R2 described above.

  The control unit 105 is associated with a partial area 151 of the entire display area of the display unit 101 according to the operation of both hands of the operator 500 determined based on the situation information obtained by the acquisition unit 102, The selection area R1 for accepting the selection operation of the operator 500 in the area 151 and the determination area R2 for accepting the determination operation of the operator 500 are separately arranged on the input instruction area. In this embodiment, as an example, since the acquisition unit 102 is the camera 14, the control unit 105 arranges the selection region R1 and the determination region R2 based on the image captured by the camera 14, that is, the situation information. .

[Operation of information input device 1]
Hereinafter, the operation of the information input apparatus 1 will be described with reference to FIGS. 1 and 2 and FIGS. FIG. 8 is a flowchart showing an operation example of the information input apparatus 1.

  In FIG. 8, the CPU 11 (area setting unit 104) of the information input apparatus 1 sets an input instruction area for the operator 500 to give an input instruction (S1). This input instruction area is stored in the ROM 12 as coordinate data, for example.

  The camera (acquiring unit 102) 14 captures (acquires) a situation as shown in FIG. 2, that is, a situation where the operator 500 gives an input instruction (S2).

  CPU11 (control part 105) distinguishes and arrange | positions selection area | region R1 and determination area | region R2 based on the image (situation information) obtained with the camera 14. FIG. In this case, when the CPU 11 determines that there is a hand pattern of the selection area R1 in the image, the CPU 11 arranges the selection area R1 on the hand side and arranges the determination area R2 on the other hand side.

  For example, in the example of FIG. 2, the selection area R1 is arranged in the input instruction area on the right hand 503 side, and the determination area R2 is arranged in the input instruction area on the left hand 502 side. Since the selection area R1 described above is associated with a part of the entire display area 151 of the display device 11 (see FIG. 6), the operator 500 in FIG. The selection operation is performed with the right hand 503, for example. FIG. 6A shows an example in which the pointer 40 in the area 151 is moved by moving the index finger of the right hand 503 as the selection operation.

  Note that the positional relationship between the regions R1 and R2 arranged in the input instruction region is associated with, for example, coordinate data.

  CPU11 (control part 105) performs the display based on the input instruction by the operator 500 (S4). For example, in the example of FIG. 6A, the pointer 40 is moved clockwise by the selection operation on the pointer 40. In this case, the CPU 11 (the control unit 105) performs a selection operation associated with the movement of the index finger of the operator in the selection area R1, based on an image from the camera 14 that captures a part of the entire display area 151. As a result, the pointer 40 is moved.

  In FIG. 8, the CPU 11 (control unit 105) determines whether or not the hand pattern is in the selection region R <b> 1 based on the image from the camera 14. The selection area R1 and the determination area R2 may be replaced with each other (S3), and display based on an input instruction may be performed (S4). For example, FIGS. 6 (a) and 6 (b) show an example in which the selection region R1 and the determination region R2 are exchanged by movement of both hands.

  As described above, according to the information input device 1 of the present embodiment, the operator 500 moves the hand, and the selection operation is performed by setting the hand pattern of the selection region R1 with one of the hands. Here, since the selection operation can be performed on a part of the entire display area associated with the selection area R1, the angle of view of the camera 14 is set so as to capture the selection area R1. be able to. Thereby, in the camera 14 of the present embodiment, the angle of view can be narrowed unlike the conventional camera configured to capture the entire display area. Further, in this information input device 1, since the selection operation of the operator 500 is determined based on the above-described image from the camera 14 with a narrow angle of view, image recognition for recognizing the movement of the hand of the operator 500 is required. The target range is narrowed and the accuracy of image recognition is increased. For example, when the display device 15 is a horizontal wide display (horizontal: vertical = 16: 9, etc.), the area for gesture operation to be recognized can be narrowed, so that the information input device 1 can obtain an image obtained from the camera 14. Recognition accuracy can be increased. In other words, the information input device 1 can recognize the movement of the hand of the operator 500 more accurately. Therefore, the selection operation by the operator 500 can be performed more accurately.

  Further, in the information input device 1 of the present embodiment, the selection area R1 is associated with a partial area 151 of the entire display area, and therefore, for example, the pointer 40 is moved to the entire display area by the gesture operation of the operator 500. It is possible to move within a part of the area 151. At this time, the result recognized based on the image obtained from the camera 14 can be reflected in the operation of the pointer 40 by narrowing down to a partial area 151 instead of the entire display area. Therefore, the information input device 1 can appropriately perform an operation on the object performed based on the gesture operation of the operator 500.

  Furthermore, a gesture as a pointing or determination operation with respect to a selection target is performed by switching the left and right regions, thereby preventing the operator 500 from moving only with one hand. For this reason, not only reduction of physical fatigue can be realized, but also a pointing operation can be performed with a hand / finger close to the pointing object.

  In addition, by making the region associated with the selection region R1 a part of the entire display region, the selection region R1 is difficult to enter from the view of the operator 500. Therefore, the camera 14 can easily capture the shape of the hand of the operator 500, and the information input device 1 can easily perform image recognition. Furthermore, the viewing angle of the mounted camera 14 can be minimized, and a camera for a general camera conference or the like that is not a wide angle can be used.

  Next, a modified example of the information input device 1 of the present embodiment will be described.

(Modification 1)
In the above, with reference to FIG. 2, the case has been described in which a single camera 14 captures an image of a situation where an operator gives an input instruction. However, a plurality of cameras may be provided.

  FIG. 9 is a diagram illustrating a modification in the case where the information input device 1 includes four cameras. In the example of FIG. 9, four cameras 14 </ b> A, 14 </ b> B, 14 </ b> C, and 14 </ b> D are attached to the display device 14. With this configuration, the accuracy of image recognition can be increased.

(Modification 2)
The positions of the selection area R1 and the determination area R2 can be freely set. For example, FIG. 10 shows a modification in which the selection area R1 and the determination area R2 are set on the desk. By doing in this way, the burden of the input instruction by the hand of the operator 500 is reduced.

(Modification 3)
The hand pattern of the selection region R1 can be changed. For example, a thumb can be put up, or a hand can be closed.

1 Information input device 11 CPU
14 camera 15 display device 101 display unit 102 acquisition unit 104 region setting unit 105 control unit 500 operator R1 selection region R2 determination region

Claims (6)

A display unit comprising a display area;
An area setting unit for setting an input instruction area for an operator to input instructions;
An acquisition unit for acquiring the status of the operator for which the input instruction is performed;
In response to the movement of both hands of the operator determined based on the acquired situation information, the operation is performed in a part of the display area and is associated with a part of the whole area. A control unit that distinguishes and arranges a selection region for accepting an operator's selection operation and a decision region for accepting the operator's decision operation on the input instruction region;
Including
The control unit arranges the selection area and the determination area in accordance with a pattern of the finger of the operator recognized from a change in movement of both hands of the operator. . The information input device according to claim 1, wherein the partial area is set to be larger than an area equally divided into two at the center of the display area. When the acquisition unit is an imaging unit that captures the situation of the operator, the imaging unit sets an angle of view so as to capture the selected region associated with the partial region, and the imaging unit Acquired images as the situation information,
Wherein the control part, based on the image from the imaging unit, the information input device according to claim 1 or 2 is configured to determine the operator's selecting operation in said selection area. Obtaining a situation in which the operator gives input instructions;
In response to the movement of both hands of the operator determined based on the acquired situation information, the operation is performed in a part of the entire display area and is associated with a part of the display area. Distinguishing and arranging a selection area for accepting an operator's selection operation and a decision area for accepting the operator's decision operation;
Including
The information input is characterized in that the step of arranging arranges the selection area and the determination area in accordance with a pattern of the finger of the operator recognized from a change in movement of both hands of the operator. Method. 5. The information input method according to claim 4 , wherein the partial area is set to be larger than an area equally divided into two at the center of the display area. A program for causing a computer to execute the information input method according to claim 4 or 5 .

Images