JP2024092095A - Operation Input System - Google Patents

Abstract

To provide an operation input system allowing an operator to perform input operation accurately by a finger.SOLUTION: An operation input system according to the present invention includes: a first sensor device which has an annular base body being wearable to a first finger of an operator, and outputs a first signal according to a motion of the first finger; a second sensor device which has an annular base body being wearable to a second finger different from the first finger and outputs a second signal according to a motion of the second finger; and a control device which determines a relative finger motion of the first finger and the second finger on the basis of the first signal and the second signal.SELECTED DRAWING: Figure 1

Description

The present invention relates to an operation input system.

Patent Document 1 discloses a smart interface ring that has a pressure-sensitive touch pointer and a three-axis gyro sensor and is worn on one of the operator's fingers. This smart interface ring captures the movement of the fingertip based on the angular velocity of the finger detected by the three-axis gyro sensor, enabling operation input for an application.

JP 2018-21270 A

The ring described in Patent Document 1 is configured to detect only the angular velocity of one finger of a hand in a specified posture to determine the movement, so there is an issue that the finger movement of the operator cannot be accurately determined, resulting in low accuracy of input operations.

Therefore, in consideration of the problems in the conventional technology described above, the present invention aims to provide an operation input system that allows an operator to perform input operations with finger movements with high accuracy.

According to one aspect of the present invention, there is provided an operation input system including a first sensor device having an annular base that can be attached to a first finger of an operator and that outputs a first signal corresponding to the movement of the first finger, a second sensor device having an annular base that can be attached to a second finger different from the first finger and that outputs a second signal corresponding to the movement of the second finger, and a control device that determines the relative finger movements of the first finger and the second finger based on the first signal and the second signal.

According to another aspect of the present invention, there is provided an operation input system including a first sensor device attached to a first finger of an operator and detecting a first signal corresponding to a movement of the first finger, a second sensor device attached to a second finger different from the first finger and detecting a second signal corresponding to a movement of the second finger, and a control device that controls the first sensor device and the second sensor device, respectively. The control device detects the relative finger movements of the first finger and the second finger based on the first signal and the second signal, and, on condition that it has determined that the finger movement is a predetermined first gesture, determines whether the finger movement performed following the first gesture is a predetermined second gesture, and, if it has determined that the finger movement is the second gesture, outputs a command corresponding to the second gesture.

The present invention provides an operation input system that allows an operator to perform input operations with finger movements with high accuracy.

1 is a block diagram showing an example of a hardware configuration of a device constituting an operation input system according to a first embodiment. 3A to 3C are diagrams illustrating an example of how the first sensor device and the second sensor device according to the first embodiment are attached to the hands of an operator. 2 is a cross-sectional view showing an example of an internal structure of a first sensor device according to the first embodiment. FIG. 2 is a cross-sectional view showing an example of an internal structure of a first sensor device according to the first embodiment. FIG. FIG. 2 is a functional block diagram showing an example of functions of a control device according to the first embodiment. 5A to 5C are diagrams illustrating a change over time in the angular velocity of the thumb detected by the first sensor device according to the first embodiment and an extraction period for a peak value. 5 is a diagram showing a change over time in angular velocity of the thumb detected by the first sensor device according to the first embodiment. FIG. 10 is a diagram showing a change over time in the angular velocity of the index finger detected by the second sensor device according to the first embodiment. FIG. 4 is a timing chart showing processes and execution times executed in the control device according to the first embodiment. 4 is a flowchart illustrating an example of a process executed in the operation input system according to the first embodiment. FIG. 11 is a block diagram showing an example of an overall configuration of an operation input system according to a second embodiment. FIG. 11 is a functional block diagram showing an example of functions of a control device according to a second embodiment. 13A and 13B are diagrams illustrating a relationship between finger motions and finger state transitions in the operation input system according to the second embodiment. 13 is a diagram showing a combination of peak values of angular velocity of the thumb used in determining each gesture in the operation input system according to the second embodiment. FIG. 13 is a diagram showing a combination of peak values of angular velocity of the index finger used in determining each gesture in the operation input system according to the second embodiment. FIG. 13 is a top view illustrating a plurality of gestures detected in the operation input system according to the second embodiment. FIG. FIG. 17 is a side view illustrating the gesture shown in FIG. 16 . FIG. 17 is a side view illustrating the gesture shown in FIG. 16 . 10 is a flowchart illustrating an example of a process executed in the operation input system according to the second embodiment. FIG. 11 is a diagram showing an example of a screen displayed on a control device according to a second embodiment. FIG. 13 is a block diagram showing an example of a hardware configuration of a device constituting an operation input system according to a third embodiment. FIG. 11 is a perspective view showing an example of smart glasses according to a third embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. Elements having common functions throughout the drawings are given the same reference numerals, and duplicate descriptions may be omitted or simplified.

[First embodiment]
FIG. 1 is a block diagram showing an example of a hardware configuration of devices constituting an operation input system 1 according to this embodiment.

As shown in FIG. 1, the operation input system 1 includes a first sensor device 10, a second sensor device 20, and a control device 30. In this embodiment, the first sensor device 10 and the second sensor device 20 are connected to the control device 30 via a network.

The first sensor device 10 has a ring-shaped base that can be attached to the first finger of an operator, and outputs a first signal corresponding to the movement of the first finger. In this embodiment, the first sensor device 10 outputs detection data of the angular velocity of the first finger (hereinafter referred to as the "first angular velocity") as a first signal corresponding to the movement of the first finger. Specifically, the first sensor device 10 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

As shown in FIG. 1, the first sensor device 10 includes an MCU (Micro Controller Unit) 101, a sensor module 102, a feedback module 103, a wireless communication device 104, and a battery 105. Each part of the first sensor device 10 is connected by wiring on a circuit board (not shown).

MCU101 is an embedded microprocessor that integrates a CPU (Central Processing Unit), RAM (Random Access Memory), ROM (Read Only Memory), I/O devices, etc. into a single integrated circuit. MCU101 is primarily used to control electronic devices.

The sensor module 102 has a gyro sensor that detects angular velocity. The gyro sensor of this embodiment is a three-axis gyro sensor that detects angular velocity around the mutually orthogonal X-axis, Y-axis, and Z-axis. In addition to the gyro sensor, the sensor module 102 may further include sensors such as an acceleration sensor and a magnetic sensor. When an acceleration sensor is used, the actual acceleration is calculated by removing the gravity component from the detected acceleration value.

The sensor module 102 applies a voltage proportional to the detected angular velocity to the MCU 101. The MCU 101 A/D converts the voltage applied from the sensor module 102 into digital data. The MCU 101 then transmits the detected angular velocity data to the control device 30 via the wireless communication device 104, which will be described later.

The feedback module 103 notifies the operator who is wearing the first sensor device 10 on his or her thumb based on the control information received from the control device 30. Details of the feedback module 103 will be described later.

The wireless communication device 104 is a communication interface based on a standard such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G, or 5G, and is a module for performing wireless communication with other devices. The wireless communication device 104 transmits the detection data of the first angular velocity to the control device 30 by wireless communication.

The battery 105 is a device that supplies power to each of the MCU 101, the sensor module 102, the feedback module 103, and the wireless communication device 104.

The second sensor device 20 has a ring-shaped base that can be attached to a second finger different from the operator's first finger, and outputs a second signal corresponding to the movement of the second finger. The second sensor device 20 of this embodiment outputs detection data of the angular velocity of the second finger (hereinafter referred to as the "second angular velocity") as a second signal corresponding to the movement of the second finger. Specifically, the second sensor device 20 transmits the detection data of the second angular velocity to the control device 30 via wireless communication.

In this embodiment, the first finger is the thumb. The second finger is preferably the finger adjacent to the first finger, and is therefore the index finger. The setting information in the first sensor device 10 and the second sensor device 20 is set to match the thumb and index finger of the person wearing the sensor device.

As shown in FIG. 1, the second sensor device 20 includes an MCU 201, a sensor module 202, a feedback module 203, a wireless communication device 204, and a battery 205. The hardware configuration of the second sensor device 20 is similar to that of the first sensor device 10, so a description of each device will be omitted.

The control device 30 is a computer device that performs calculations, control, and storage. Examples of the control device 30 include a personal computer, a laptop computer, a tablet terminal, a smartphone, etc. The control device 30 determines the relative finger movements of the first finger and the second finger based on the first angular velocity and the second angular velocity.

As shown in FIG. 1, the control device 30 includes a processor 301, a RAM 302, a ROM 303, a storage 304, a communication I/F (Interface) 305, a display device 306, and an input device 307. Each device is connected to each other via a bus, wiring, a drive device, etc.

The processor 301 performs predetermined calculations according to programs stored in the ROM 303, the storage 304, etc., and has the function of controlling each part of the control device 30, the first sensor device 10, and the second sensor device 20. As the processor 301, a CPU, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), etc. may be used.

RAM 302 is a storage device that is composed of a volatile storage medium and provides temporary memory space necessary for the operation of processor 301. RAM 302 may be, for example, a D-RAM (Dynamic RAM).

ROM 303 is composed of a non-volatile storage medium and stores necessary information such as programs used in the operation of control device 30. ROM 303 may be, for example, a programmable ROM (P-ROM).

Storage 304 is a storage device that is composed of a non-volatile storage medium and stores data processed by control device 30 and programs for operating control device 30. Storage 304 is composed of, for example, a hard disk drive (HDD) or a solid state drive (SSD).

The processor 301 loads programs stored in the ROM 303, storage 304, etc. into the RAM 302 and executes them.

The communication I/F 305 is a communication interface based on standards such as Bluetooth, Ethernet (registered trademark), Wi-Fi (registered trademark), 4G or 5G, and is a module for wireless communication with the first sensor device 10 and the second sensor device 20.

The display device 306 displays moving images, still images, characters, etc. As the display device 306, a liquid crystal display, an OLED (Organic Light Emitting Diode) display, etc. are used. The operation input screen displayed on the display device 306 includes graphics such as windows, icons, buttons, etc. The operator selects a graphic representing a desired operation on the operation input screen by an operation input means using a pointing device, or an operation input means using the first sensor device 10 and the second sensor device 20, etc.

The input device 307 is a keyboard, a pointing device, etc., and is used by the operator to operate the control device 30. Examples of pointing devices include a mouse, a trackball, a touch panel, a pen tablet, etc. The display device 306 and the input device 307 may be integrally formed as a touch panel.

The hardware configuration shown in FIG. 1 is an example, and other devices may be added, or some devices may not be provided. Some devices may be replaced with other devices having similar functions. Some functions of the first embodiment may be provided by other devices via a network, or the functions of the first embodiment may be distributed and realized among multiple devices. The illustrated hardware configuration can be modified as appropriate.

Figure 2 is a diagram showing an example of how the first sensor device 10 and the second sensor device 20 according to this embodiment are attached to the fingers of an operator. In Figure 2, the first sensor device 10 is attached to the base side of the proximal phalanx of the operator's thumb F1. The second sensor device 20 is attached to the base side of the proximal phalanx of the operator's index finger F2.

In the XYZ coordinate system of the second sensor device 20, the X axis is set in the width direction of the operator's index finger F2, the Y axis is set in the extension direction of the proximal phalanx of the index finger F2, and the Z axis is set in the thickness direction of the index finger F2. The X, Y, and Z axes are mutually perpendicular. Note that, similar to the second sensor device 20, in the XYZ coordinate system of the first sensor device 10, the X axis is set in the width direction of the operator's thumb F1, the Y axis is set in the extension direction of the proximal phalanx of the thumb F1, and the Z axis is set in the thickness direction of the thumb F1.

Figures 3 and 4 are cross-sectional views showing an example of the internal structure of the first sensor device 10 according to this embodiment. Note that the internal structure of the second sensor device 20 is similar to the internal structure of the first sensor device 10, so the first sensor device 10 will be described below as a representative example.

In FIG. 3, an example is shown in which a tightening mechanism 103A is used as the feedback module 103 in the first sensor device 10. The tightening mechanism 103A expands or contracts based on control information from the control device 30. Examples of the tightening mechanism 103A include an airbag or a dielectric member (e.g., e-Rubber, a next-generation rubber material manufactured by Toyoda Gosei Co., Ltd.) that can expand or contract depending on an applied voltage. When the tightening mechanism 103A expands, it deforms into a protrusion toward the operator's finger, and a pressing force is applied to the operator's finger. As a result, the control device 30 causes the feedback module 103 to execute feedback on the operation input from the operator. The feedback is performed to notify the operator that the input operation by the operator has been accepted and a specific command has been executed.

In FIG. 3, the tightening mechanism 103A is formed in a ring shape, similar to the base 11, but the shape of the tightening mechanism 103A is not limited to this. The tightening mechanism 103A may be formed in a sheet shape, for example. Also, the number of tightening mechanisms 103A is one, but is not limited to this.

When multiple tightening mechanisms 103A are provided, there will be multiple locations within the device that can expand or contract. Therefore, by switching the tightening mechanism 103A to be expanded or contracted within the first sensor device 10 according to time, feedback in a clockwise or counterclockwise direction, etc., is also possible. In other words, it is possible to further increase the variety of feedback methods.

The first sensor device 10 also includes three sensor modules 102. The three sensor modules 102 are arranged at approximately equal intervals around the circumference of the base body 11 in a cross-sectional view. Sensors such as gyro sensors detect different changes in the value of angular velocity depending on the mounting position. For this reason, by providing the three sensor modules 102 at a distance from each other, angular velocity can be detected at different positions. As a result, the control device 30 can determine the mounting state of the first sensor device 10 on the thumb F1 with high accuracy based on the multiple angular velocities. The same is true for the second sensor device 20.

On the other hand, FIG. 4 shows an example in which three vibration mechanisms 103B are used as the feedback module 103 in the first sensor device 10. The number of sensor modules 102 is three, the same as in FIG. 3.

The vibration mechanism 103B vibrates based on control information from the control device 30. The vibration mechanism 103B is composed of, for example, a piezoelectric element (piezo element) that can expand or contract depending on an applied voltage, and an electromagnetic drive unit such as a vibration motor that can vibrate. A piezo element is a passive element that performs micro-motion control and detection by utilizing the piezoelectric effect and inverse piezoelectric effect that occur in dielectric materials such as crystal or quartz. It has a simple structure that does not require gears or motors for operation, and is therefore a smaller element than other micro-motion mechanism elements.

The vibration mechanism 103B does not have to be provided adjacent to the sensor module 102, and may be provided between multiple sensor modules 102. When the vibration mechanism 103B vibrates, the vibration is transmitted to the operator's finger. As a result, the control device 30 causes the feedback module 103 to perform feedback on the operation input from the operator.

In FIG. 4, the number of vibration mechanisms 103B is the same as the number of sensor modules 102, but the number of vibration mechanisms 103B is not limited to this. The magnitude and period of vibration in each module may be changed.

When multiple vibration mechanisms 103B are provided, there will be multiple locations within the device that can vibrate. By switching the vibration mechanism 103B that vibrates within the first sensor device 10 according to time, feedback in a clockwise or counterclockwise direction, etc., is also possible. In other words, it is possible to further increase the variety of feedback methods.

Figure 5 is a functional block diagram showing an example of the functions of the control device 30 according to this embodiment. The control device 30 includes a calibration unit 30A, an angular velocity calculation unit 30B, a peak value extraction unit 30C, a command issuing unit 30D, a feedback control unit 30E, a display control unit 30F, and an input unit 30G.

When the first sensor device 10 and the second sensor device 20 are attached to the operator's finger and activated for the first time, the calibration unit 30A performs calibration on the detection data of the angular velocity detected by the sensor modules 102 and 202. The calibration unit 30A outputs the correction value (offset value) of the angular velocity calculated by the calibration to the angular velocity calculation unit 30B.

The angular velocity calculation unit 30B calculates the angular velocity of the operator's thumb F1 and index finger F2 based on the angular velocity detection data acquired by wireless communication from the first sensor device 10 and the second sensor device 20 and the correction value calculated by the calibration unit 30A. The angular velocity calculation unit 30B outputs the calculated angular velocity to the peak value extraction unit 30C. The angular velocity calculation process in the angular velocity calculation unit 30B is repeatedly executed each time angular velocity detection data is received from the first sensor device 10 and the second sensor device 20.

The peak value extraction unit 30C extracts peak values from the angular velocities within a predetermined extraction period. The peak value extraction unit 30C outputs the extracted peak values to the command issuing unit 30D.

Figure 6 is a diagram showing an example of the time change in angular velocity detected by the first sensor device 10 according to this embodiment and the extraction period of the peak value. In Figure 6, as shown by the dashed lines, the positive threshold value THp of the angular velocity is set to +20 rad/sec, and the negative threshold value THm is set to -20 rad/sec. The absolute values of the threshold values THp and THm are equal. The extraction period T is set to time t2, when a certain amount of time has elapsed from time t1 when at least one of the angular velocities around the X-axis, Y-axis, and Z-axis of two fingers becomes equal to or exceeds a predetermined threshold value. However, the values of the threshold values THp and THm and the method of setting the extraction period T are not limited to this.

The command issuing unit 30D determines the relative finger movement of the operator's two fingers based on the peak values extracted within the extraction period, and outputs a command that is pre-associated with the finger movement. The command issuing unit 30D detects at least two types of finger movements. When the command issuing unit 30D outputs a command, a process corresponding to the command is executed in one of the devices that make up the operation input system 1.

The feedback control unit 30E transmits control information to the MCU 101 of the first sensor device 10 and the MCU 201 of the second sensor device 20, and causes the feedback modules 103 and 203 to execute a predetermined feedback process.

The display control unit 30F controls the display of the operation input screen displayed on the display device 306 based on the input operation by the input device 307 received by the input unit 30G and the input operation by the first sensor device 10 and the second sensor device 20.

The input unit 30G accepts information input by a user, such as an operator or administrator, using the input device 307 to operate the control device 30.

Figure 7 is a diagram showing the change over time in the angular velocity of the thumb F1 detected by the first sensor device 10 according to this embodiment. Meanwhile, Figure 8 is a diagram showing the change over time in the angular velocity of the index finger F2 detected by the second sensor device 20 according to this embodiment. Figures 7 and 8 show an extraction period A from time t1 to t2, an extraction period B from time t3 to t4, and an extraction period C from time t5 to t6.

The threshold values THp and THm for the angular velocity of the thumb F1 in FIG. 7 are different from the threshold values THp and THm for the angular velocity of the index finger F2 in FIG. 8. In this way, it is preferable to set the threshold values THp and THm to optimal values for each finger. The control device 30 determines the relative finger movements of the thumb F1 and index finger F2 in each of the extraction periods A, B, and C based on the peak values of the angular velocities of the two fingers around the X-axis, Y-axis, and Z-axis in the extraction periods A, B, and C and the threshold values THp and THm.

As shown in FIG. 7, in extraction period A, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is greater than the positive threshold value THp. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the negative side, and the peak value on the negative side is greater than the negative threshold value THm. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp.

As shown in FIG. 8, in extraction period A, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold value THp. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the positive side, and the peak value on the positive side is greater than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on both the positive and negative sides, and the peak value on the positive side is less than the threshold value THp, and the peak value on the negative side is greater than the threshold value THm.

When analyzing the change in angular velocity in Figures 7 and 8 for extraction period A, it can be seen that the thumb F1 moves significantly in the +X direction around the X axis. It can also be seen that in extraction period A, the index finger F2 moves significantly in the +Y direction around the Y axis. As a result, a finger motion in which the thumb F1 is raised above the index finger F2 is detected in extraction period A.

As shown in FIG. 7, in extraction period B, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

As shown in FIG. 8, in extraction period B, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is greater than the threshold value THm. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp.

Analyzing the change in angular velocity in Figures 7 and 8 for extraction period B shows that the thumb F1 moves significantly in the -X direction around the X axis. It can also be seen that in extraction period B, the index finger F2 moves significantly in the -Y direction around the Y axis. As a result, a finger movement in which the thumb F1 touches the index finger F2 is detected in extraction period B.

As shown in FIG. 7, in extraction period C, the angular velocity X_1 of the thumb F1 around the X-axis is displaced on the positive side, and the peak value on the positive side is smaller than the threshold value THp. Also, the angular velocity Y_1 of the thumb F1 around the Y-axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_1 of the thumb F1 around the Z-axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

As shown in FIG. 8, in extraction period C, the angular velocity X_2 of the index finger F2 around the X axis is displaced on the negative side, and the peak value on the negative side is smaller than the threshold value THm. The angular velocity Y_2 of the index finger F2 around the Y axis is displaced on the positive side, and the peak value on the positive side is larger than the threshold value THp. The angular velocity Z_2 of the index finger F2 around the Z axis is displaced on the negative side, and the peak value on the negative side is larger than the threshold value THm.

Analyzing the change in angular velocity in Figures 7 and 8 for extraction period C shows that the thumb F1 moves significantly in the +Y direction around the Y axis. It is also clear that in extraction period C, the index finger F2 moves significantly in the -X direction around the X axis and in the +Y direction around the Y axis. As a result, in extraction period C, a finger motion is detected in which the thumb F1 maintains contact with the index finger F2 while sliding to the right along the side of the index finger F2.

Figure 9 is a timing chart showing the processes and execution times executed by the control device 30 according to this embodiment. Here, the correspondence between the extraction periods A, B, and C shown in Figures 7 and 8 and the command issuing process, feedback destination determination process, drive control process for the feedback module, and display control process for the operation input screen is shown.

The first command issuance process in the command issuing unit 30D is executed during extraction period A from time t1 to t2. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t2 to t21. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t21 to t22.

The second command issuing process in the command issuing unit 30D is executed during extraction period B from time t3 to t4. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t4 to t41. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t41 to t42.

The third command issuance process in the command issuing unit 30D is executed during extraction period C from time t5 to t6. Next, the feedback destination determination process in the feedback control unit 30E is executed during the period from time t6 to t61. The drive control process of the feedback modules 103, 203 in the feedback control unit 30E and the screen display process in the display control unit 30F are executed synchronously during the period from time t61 to t62.

Figure 10 is a flowchart showing an example of a process executed by the control device 30 according to this embodiment. This process is executed repeatedly, for example, while the operation input screen is displayed.

First, the control device 30 acquires angular velocity detection data from the first sensor device 10 and the second sensor device 20 via wireless communication (step S101).

Next, the control device 30 determines whether or not the first sensor device 10 and the second sensor device 20 are being started for the first time (step S102). If the control device 30 determines that the first sensor device 10 and the second sensor device 20 are being started for the first time (step S102: YES), the process proceeds to step S103. On the other hand, if the control device 30 determines that the first sensor device 10 and the second sensor device 20 are not being started for the first time (step S102: NO), the process proceeds to step S105.

In step S103, the control device 30 performs calibration based on the angular velocity detection data acquired from the first sensor device 10 and the second sensor device 20, and calculates a correction value for the angular velocity. After that, the process proceeds to step S104.

In step S104, the control device 30 determines the mounting state of the sensor based on the angular velocity detection data. Then, the process proceeds to step S105.

As shown in FIG. 3, the first sensor device 10 includes multiple sensor modules 102. Therefore, the control device 30 can determine the attachment state of the first sensor device 10 on the thumb F1 based on multiple detection data detected by the multiple sensor modules 102.

Similarly, the second sensor device 20 includes multiple sensor modules 202. The control device 30 can determine the attachment state of the second sensor device 20 on the index finger F2 based on multiple detection data detected by the multiple sensor modules 202.

In step S105, the control device 30 calculates the angular velocities of the thumb F1 and index finger F2, respectively, based on the angular velocity detection data and the angular velocity correction value. The angular velocities of the thumb F1 and index finger F2 are calculated for three axes: the X-axis, the Y-axis, and the Z-axis.

Next, the control device 30 determines whether any of the absolute values of the calculated angular velocities is equal to or greater than a predetermined threshold value (step S106).

If the control device 30 determines that the absolute value of any of the angular velocities is equal to or greater than the predetermined threshold (step S106: YES), the process proceeds to step S107. On the other hand, if the control device 30 determines that the absolute values of all of the angular velocities are less than the predetermined threshold (step S107: NO), the process returns to step S101.

In step S107, the control device 30 determines an extraction period for the angular velocity and extracts peak values from the angular velocity within the extraction period. Peak values are extracted for each of the three axes, the X-axis, the Y-axis, and the Z-axis. If the angular velocity fluctuates within the extraction period, multiple peak values may be extracted in the same direction. In such a case, the control device 30, for example, sequentially updates the peak values stored in the memory area.

Next, the control device 30 determines the relative finger motion of the two fingers based on the combination of the peak values of the angular velocities of the two fingers extracted within the extraction period (step S108). The finger motion determination is started when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 exceeds a predetermined threshold, so the calculation load is reduced.

In step S108, for example, finger movements are determined, such as the thumb F1 and index finger F2 moving apart, the thumb F1 and index finger F2 coming into contact with each other, and the thumb F1 moving up and down, left and right, or back and forth along the index finger F2 while the thumb F1 and index finger F2 are in contact with each other.

Next, when the control device 30 determines the command corresponding to the relative movement of the two fingers, it outputs the command (step S109). As a result, the control device 30 executes the process specified by the command. Examples of commands include a command to move the position of the cursor or pointer on the operation input screen up, down, left, or right, a command to cancel the operation input performed immediately before, a command to confirm the operation input, etc.

Next, the control device 30 determines the feedback destination in conjunction with the execution of the issued command (step S110). In this embodiment, the first sensor device 10, the second sensor device 20, and the display device 106 can be the feedback destinations.

Next, the control device 30 transmits control information to the feedback modules 103 and 203 that are the determined feedback destinations, and drives the feedback modules 103 and 203 (step S111).

Next, the control device 30 updates the operation input screen displayed on the display device 106 (step S112). For example, the control device 30 changes the coordinates of the pointer on the operation input screen, or changes the display state of the selected menu. In this way, the control device 30 notifies the GUI (display device) in synchronization with the notification to the feedback module. Then, the process ends.

Then, the control device 30 determines whether or not the operator has instructed the operation input to end on the operation input screen (step S113). Here, if the control device 30 determines that the operator has instructed the operation input to end (step S113: YES), the process ends. On the other hand, if the control device 30 determines that the operator has not instructed the operation input to end (step S113: NO), the process returns to step S101.

Conventionally, input means using gestures or controllers, input means based on gaze, and input means based on voice are known as input means that can replace mouse operation. However, with input means using gestures or controllers, the operation input motions are large, and the operation status can be seen by others. Also, with input means based on gaze, the gaze position fluctuates, making stable operation difficult. And with input means based on voice, the voice giving instructions on operation can be heard by others. In view of these problems, there has been a demand for a pointing device that is difficult for others to recognize and has excellent operability.

In contrast, according to the operation input system 1 of this embodiment, a first sensor device 10 and a second sensor device 20, which are ring-shaped devices equipped with a gyro sensor, are attached to the thumb (first finger) F1 and the index finger (second finger) F2, respectively. The first sensor device 10 and the second sensor device 20 then transmit the angular velocities of the two fingers to the control device 30 via wireless communication. The control device 30 detects the relative finger movements of the two fingers using the angular velocities of the thumb F1 and the index finger F2.

This allows commands corresponding to finger movements to be determined and output, allowing operation input to be performed freely and with high accuracy. In addition, because operation input can be performed simply by moving the thumb F1 and index finger F2, it is difficult for others to know that the operator is performing operation input, which is also beneficial from the standpoint of improving information security and protecting privacy.

Furthermore, in conventional systems where a ring-shaped device was worn on only one finger, it was necessary to change the direction of the arm in order to point to a specific position with a finger. In contrast, the operation input system 1 according to this embodiment is configured to determine the relative movement of the two fingers by wearing ring-shaped devices equipped with gyro sensors on the thumb F1 and index finger F2, respectively. This makes it possible to point with the arm in any orientation.

In addition, some conventional ring-shaped devices are known to be equipped with a gyro sensor for detecting motion. However, the information that can be detected varies depending on the mounting position of the gyro sensor, so the ring-shaped device always needs to be worn in a fixed direction. In contrast, according to the operation input system 1 of this embodiment, the ring-shaped device (first sensor device 10 and second sensor device 20) is equipped with multiple sensor modules at different positions within the ring. Since the mounting state of the ring-shaped device can be determined based on multiple detection data from multiple sensor modules, the ring-shaped device can be worn in any position on each finger.

[Second embodiment]
The operation input system 2 according to the second embodiment will be described below. The differences from the first embodiment will be mainly described below, and descriptions of common parts will be omitted or simplified.

FIG. 11 is a block diagram showing an example of the hardware configuration of the operation input system 2 according to this embodiment. The hardware configuration of the operation input system 2 is common to the hardware configuration of the operation input system 1 according to the first embodiment.

Figure 12 is a functional block diagram showing an example of the functions of the control device 30 according to this embodiment. Unlike the first embodiment, the control device 30 of this embodiment further includes a gesture detection unit 30H.

The gesture detection unit 30H detects a specific finger motion from among arbitrary finger motions made by the thumb F1 and index finger F2 based on the angular velocity of the thumb F1 and index finger F2. In this embodiment, the specific finger motion that the gesture detection unit 30H detects from among arbitrary finger motions is called a "gesture." Note that gestures are not limited to single finger motions. There may also be cases where multiple finger motions performed consecutively within a certain period of time are defined as one gesture. It is assumed that the correspondence between the multiple gestures and the commands issued for each gesture is defined in advance. The command issuing unit 30D of this embodiment outputs a command corresponding to the specific gesture detected by the gesture detection unit 30H.

Figure 13 is a diagram showing the relationship between finger motions and finger state transitions in the operation input system 2 according to this embodiment. Here, "thumb up", "thumb down", "thumb down", "stationary", and "moving" are shown as finger motions. Also, "initial (ST0)", "left (ST1)", "right (ST2)", "front (ST3)", "back (ST4)", "down (ST5)", "release (ST6)", and "contact (ST7)" are shown as finger states.

"Initial state (ST0)" is a state in which the operator lightly closes his or her hand and places the thumb F1 midway between the first and second joints of the index finger F2. Hereinafter, "initial state (ST0)" will be referred to as the initial state ST0.

"Left (ST1)" is the state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved leftward along the side of the index finger F2. Hereinafter, "left (ST1)" will be referred to as state ST1.

"Right (ST2)" is the state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved to the right along the side of the index finger F2. Hereinafter, "right (ST2)" is referred to as state ST2.

"Forward (ST3)" is a state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, while the tip of the thumb F1 is protruding outward from the proximal joint of the index finger F2. Hereinafter, "forward (ST3)" will be referred to as state ST3.

"Back (ST4)" is the state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is moved to the side of the proximal joint of the index finger F2. Hereinafter, "back (ST4)" will be referred to as state ST4.

"Down (ST5)" is a state in which the thumb F1 and index finger F2 are in contact with each other in the initial state ST0, and the tip of the thumb F1 is pressed down to a position below the index finger F2. Hereinafter, "down (ST5)" is referred to as state ST5. In state ST5, the thumb F1 is located between the index finger F2 and the middle finger F3.

"Release (ST6)" is the state when the thumb F1 and index finger F2, which had been in contact, are separated and maintained in this state for a certain period of time. Hereinafter, "release (ST6)" will be referred to as the release state ST6.

"Contact (ST7)" is a state in which the thumb F1 and index finger F2, which were in the released state ST6, are brought into contact with each other. In the following, "Contact (ST7)" will be referred to as state ST7. State ST7 is an intermediate state when changing from the released state ST6 to the initial state ST0.

The control device 30 (gesture detection unit 30H) determines the state of the thumb F1 and index finger F2 based on the change in angular velocity. When the state transitions from the release state ST6 to the contact state ST7 and the contact state ST7 continues for a certain period of time, the control device 30 determines that the current state is the initial state ST0.

In addition, in this embodiment, the first gesture and the second gesture are defined as follows.
(1) First Gesture The first gesture is a motion of moving the thumb F1 and index finger F2 from any one of states ST0 to ST5 to the initial state ST0 via states ST6 and ST7. In other words, the first gesture is a motion of lifting the thumb F1 and lowering the thumb F2.
(2) Second Gesture The second gesture is one of the following two types of actions (A) and (B).
(A) An action of putting the thumb F1 and index finger F2 into one of states ST1 to ST5, starting from the initial state ST0.
(B) A series of actions in which a similar action (ST0→ST6→ST7→ST0) is performed within a certain period of time immediately following the action (ST0→ST6→ST7→ST0) that is the first gesture.
The action (B) itself is the same as the first gesture, but if the action (B) is performed immediately after and within a certain period of time after the first gesture, it is treated as the second gesture. In other words, the action (B) is a double-tap gesture that combines multiple actions.

The first gesture and the second gesture are defined to be different from the operator's normal finger movements. This allows the first gesture and the second gesture to be easily detected from any finger movements. In addition, it is preferable that the first gesture is a characteristic finger movement different from the second gesture. This allows the control device 30 to easily distinguish between the first gesture and the second gesture. In this embodiment, the first gesture is a large up and down movement of the thumb, and the second gesture is a front-back and left-right movement of the index finger within a plane or a small downward movement based on the same plane.

Figure 14 is a diagram showing combinations of peak values of angular velocity of the thumb F1 used in determining each gesture in the operation input system 2 according to this embodiment. The circles ("◯") in the figure indicate the direction, out of the six directions +X, -X, +Y, -Y, +Z, and -Z, in which a peak value of angular velocity equal to or greater than a predetermined threshold value (hereinafter referred to as "peak angular velocity") is detected. The combination of directions of peak angular velocity marked with a circle indicates the characteristics of each gesture. Note that even when there is no circle, it is assumed that the angular velocity is changing very slightly.

First, we will explain the change in angular velocity of the thumb F1 during the second gesture. During the second gesture of moving the thumb F1 to the right, the angular velocity of the thumb F1 changes significantly in the three directions of +X, +Y, and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the three directions of +X, +Y, and -Z.

Similarly, during the second gesture of moving the thumb F1 to the left, the angular velocity of the thumb F1 changes significantly in two directions, -Y and +Z. The second gesture of moving the thumb F1 to the left is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, -Y and +Z.

When the second gesture involves moving the thumb F1 forward, the angular velocity of the thumb F1 changes significantly in five directions: +X, +Y, -Y, +Z, and -Z. The second gesture involves moving the thumb F1 forward, and is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the five directions: +X, +Y, -Y, +Z, and -Z.

When the second gesture involves moving the thumb F1 backward, the angular velocity of the thumb F1 changes significantly in five directions: +X, -X, +Y, -Y, and -Z. The second gesture involves moving the thumb F1 backward, and is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the five directions: +X, -X, +Y, -Y, and -Z.

When the second gesture involves tapping the thumb F1, the angular velocity of the thumb F1 changes significantly in the three directions of +X, -X, and +Y. The second gesture involves tapping the thumb F1, and is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the three directions of +X, -X, and +Y.

When the second gesture involves pushing down the thumb F1, the angular velocity of the thumb F1 changes significantly in the two directions, -X and -Z. The second gesture involves pushing down the thumb F1, and is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the two directions, -X and -Z.

Next, the change in angular velocity of the thumb F1 during the first gesture will be described. As described above, the first gesture is a series of actions that move the thumb F1 and index finger F2 from one of states ST0 to ST5 to the initial state ST0 via states ST6 and ST7. For this reason, FIG. 14 separately shows whether or not a peak angular velocity is detected in the thumb F1 during the action to move to the released state ST6 and the action to move to the initial state ST0.

When the thumb F1 and index finger F2 are put into the release state ST6, the angular velocity of the thumb F1 changes significantly in the four directions, +X, +Y, +Z, and -Z. Therefore, the action of putting the thumb F1 into the release state ST6 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of the four directions, +X, +Y, +Z, and -Z.

In addition, when the thumb F1 and index finger F2 are moved from the released state ST6 to the initial state ST0, the angular velocity of the thumb F1 changes significantly in two directions, +X and +Y. The movement of the thumb F1 and index finger F2 from the released state ST6 to the initial state ST0 is characterized in that the peak angular velocity of the thumb F1 is detected in a combination of two directions, +X and +Y.

Figure 15 is a diagram showing combinations of peak angular velocities of the index finger F2 used to determine each gesture in the operation input system 2 according to this embodiment. The circles ("◯") in the figure are the same as those in Figure 14.

First, we will explain the change in angular velocity of the index finger F2 during the second gesture. During the second gesture of moving the thumb F1 to the right, the angular velocity of the index finger F2 changes significantly in two directions, -X and -Z. The second gesture of moving the thumb F1 to the right is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of two directions, -X and -Z.

During the second gesture of moving the thumb F1 to the left, the peak angular velocity of the index finger F2 is not detected in either direction. However, the angular velocity of the index finger F2 changes slightly in both directions. During the second gesture of moving the thumb F1 to the left, no combination of peak angular velocities of the index finger F2 is detected, so there is no feature that is used for gesture determination.

During the second gesture of moving the thumb F1 forward, the angular velocity of the index finger F2 changes significantly in the three directions of -X, -Y, and -Z. The second gesture of moving the thumb F1 forward is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the three directions of -X, -Y, and -Z.

When the second gesture involves moving the thumb F1 backward, the angular velocity of the index finger F2 changes significantly in the two directions, +X and +Y. The second gesture involves moving the thumb F1 backward, and is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the two directions, +X and +Y.

During the second gesture of tapping the thumb F1, the peak angular velocity of the index finger F2 is not detected in either direction. However, the angular velocity of the index finger F2 changes slightly in both directions. During the second gesture of tapping the thumb F1, no combination of peak angular velocities of the index finger F2 is detected, and therefore no feature is used for gesture determination.

When the second gesture involves pushing down the thumb F1, the angular velocity of the index finger F2 changes significantly in the two directions, +X and +Y. The second gesture involves pushing down the thumb F1, and is characterized in that the peak angular velocity of the index finger F2 is detected in a combination of the two directions, +X and +Y.

Next, we will explain the change in angular velocity of the index finger F2 during the first gesture. In FIG. 15, similar to FIG. 14, the presence or absence of detection of peak angular velocity during the action to set the release state ST6 and the action to set the initial state ST0 are shown separately.

During the action of putting the thumb F1 and index finger F2 into the released state ST6, the peak angular velocity of the index finger F2 is not detected in either direction. However, the angular velocity of the index finger F2 changes slightly in either direction. In the first half of the action of the first gesture of putting the thumb F1 and index finger F2 into the released state ST6, the combination of peak angular velocities of the index finger F2 is not detected, so there is no feature that is used for gesture determination.

When moving from the released state ST6 to the initial state ST0, the angular velocity of the index finger F2 is not detected in any direction. However, the angular velocity of the index finger F2 changes slightly in any direction. In the latter half of the first gesture, moving from the released state ST6 to the initial state ST0, the combination of peak angular velocities of the index finger F2 is not detected, so there is no feature that is used for gesture determination.

Figure 16 is a diagram illustrating the relationship between multiple gestures detected in the operation input system 2 according to this embodiment. Eight types of gestures G0 to G6, G8 corresponding to the above-mentioned states ST0 to ST6, ST8 are shown in Figure 16. The direction of the arrow in the figure indicates the direction of transition from one gesture to another. Also, Figure 17 is a side view illustrating gesture G5. Figure 18 is a side view illustrating gesture G6.

Gesture G0 is a first gesture that changes from a state ST6 in which the thumb F1 and index finger F2 are separated to a state ST7 in which the thumb F1 is in contact with the side of the middle phalanx of the index finger F2, and then to the initial state ST0, which is maintained for a certain period of time.

Gesture G1 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the tip of the thumb F1 is moved leftward along the side of the index finger F2 while maintaining contact between the thumb F1 and index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST1 by gesture G1.

Gesture G2 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are brought into contact with each other and the tip of the thumb F1 is moved to the right along the side of the index finger F2 for a certain period of time. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST2 by gesture G2.

Gesture G3 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are in contact with each other and the tip of the thumb F1 is pushed outward beyond the back of the index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST3 by gesture G3.

Gesture G4 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are brought into contact and contracted, and the tip of the thumb F1 is moved to the side of the index finger F2. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST4 by gesture G4.

Gesture G5 is a second gesture in which, while the thumb F1 and index finger F2 are in contact with each other when they are in the initial state ST0, the tip of the thumb F1 is pressed down to contact the index finger F2 and middle finger F3 as shown in FIG. 17. The state of the thumb F1 and index finger F2 is changed from the initial state ST0 to state ST5 by gesture G5.

Gesture G6 is a second gesture in which the thumb F1 and index finger F2 are separated from an arbitrary state and maintained in that state for a certain period of time, as shown in FIG. 18. For example, the state of the thumb F1 and index finger F2 is changed by gesture G6 from the initial state ST0 or any of states ST1 to ST5 to the released state ST6.

Gesture G8 is a second gesture in which, when the thumb F1 and index finger F2 are in the initial state ST0, the thumb F1 and index finger F2 are temporarily separated, and then the thumb F1 taps the side of the index finger F2 a predetermined number of times (e.g., twice) within a predetermined time limit.

Gestures G1 to G5 and G8 are second gestures and are pre-associated with six types of commands related to operation input. Gestures G1 to G4 correspond to commands for moving the cursor or the like to the left, right, up, and down, respectively, on the operation input screen. Gesture G5 corresponds to a command for canceling the operation input performed immediately before on the operation input screen. Gesture G8 corresponds to a command for confirming the item selected or the information entered on the operation input screen.

Figure 19 is a flowchart showing an example of processing executed in the operation input system 2 according to this embodiment. The processing in Figure 19 shares steps S101 to S108 and S110 to S113 with the flowchart in Figure 10. Therefore, steps that differ from Figure 10 will be described in detail below.

In step S108, the control device 30 determines the relative finger motion of the two fingers based on the combination of peak values of the angular velocities extracted within the extraction period. The finger motion determination is started when the first angular velocity (first signal) of the thumb F1 or the second angular velocity (second signal) of the index finger F2 is equal to or greater than a predetermined threshold (step S106: YES), thereby reducing the calculation load.

After step S108, the process proceeds to step S201. In step S201, the control device 30 determines whether the finger movement using two fingers corresponds to the first gesture.

Here, if the control device 30 determines that the finger movement corresponds to the first gesture, i.e., that the finger movement is gesture G0 (step S201: YES), the control device 30 updates the detection flag for the first gesture stored in the memory area to ON (step S202), and the process returns to step S101.

The detection flag is a flag that indicates whether or not the first gesture has been detected. The detection flag can be initialized to OFF, for example, when an operation input command is issued. Note that the method of managing the detection state of the first gesture in the operation input system 1 is not limited to the method of using the detection flag.

In contrast, if the control device 30 determines that the finger movement does not correspond to the first gesture (step S201: NO), the process proceeds to step S203.

In step S203, the control device 30 determines whether the finger motion is the second gesture. If the control device 30 determines that the finger motion is the second gesture (step S203: YES), the process proceeds to step S204.

In contrast, if the control device 30 determines that the finger movement is not the second gesture (step S203: NO), the process returns to step S101.

In step S204, the control device 30 determines whether the detection flag for the first gesture is ON. That is, the control device 30 determines whether the first gesture was detected before the second gesture. Here, if the control device 30 determines that the detection flag for the first gesture is ON (step S204: YES), the process proceeds to step S205.

In contrast, if the control device 30 determines that the detection flag for the first gesture is not ON (step S204: NO), the process returns to step S101. In other words, if the control device 30 determines that the second gesture was performed before the first gesture was detected, the control device 30 does not accept the operation input by the second gesture.

In step S205, the control device 30 outputs a command corresponding to the second gesture. More specifically, when the control device 30 determines that the second gesture has been performed following the first gesture, it accepts the second gesture as an operation input and outputs a command corresponding to the second gesture. Then, the process proceeds to step S110. The processes in steps S110 to S113 are the same as those in FIG. 10.

In the case of FIG. 7 described above, the finger movement from the start time t1 of the extraction period A to the end time t4 of the extraction period B corresponds to the first gesture. Furthermore, the finger movement in the extraction period C following the first gesture corresponds to the second gesture. In the first gesture, a pair of large amplitudes on the positive side and then on the negative side occurs for the angular velocity X_1 of the thumb F1 about the X axis, and it is easy to determine whether the finger movement is a lifting of the thumb F1 in the extraction period A or a downing of the thumb F1 in the extraction period B. Furthermore, it is also easy to determine that the combination of the finger movement in the extraction period A and the finger movement in the extraction period B is the first gesture.

For example, in FIG. 7, a threshold value THp is set on the positive side and a threshold value THm is set on the negative side for the angular velocity X_1 of the thumb F1 around the X axis. Here, the control device 30 (gesture detection unit 30H) classifies the period when X_1≧THp for the angular velocity X_1 as "1", the period when THm<X_1<THp is "0", and the period when X_1≦THm is "-1". In this case, the control device 30 can easily detect the change pattern of the angular velocity of "0" → "1" → "0" → "-1" → "0" during the period from time t1 to t6. Then, the control device 30 executes the detection process of the change pattern of the angular velocity for the other angular velocities Y_1, Z_1, X_2, Y_2, and Z_2 in the same manner as for the angular velocity X_1. This allows the control device 30 to detect the second gesture performed following the first gesture with a small calculation load.

Figure 20 is a diagram showing an example of an operation input screen displayed on the display device 306 according to this embodiment. In Figure 20, multiple menus are displayed on the operation input screen, and menu M1 is shown as being selected from among them. Also, a cross key K for moving the cursor position is displayed in the upper right area of the operation input screen. Multiple indicators are provided around the cross key K. When a command to move the cursor position up, down, left, or right is issued by the operator's finger movement, the indicator for the direction corresponding to the command lights up. In Figure 20, the indicator below the cross key K is lit up in response to the command being issued to move the cursor position downward.

Conventional operation input systems based on finger gestures sometimes had trouble detecting the boundary between one gesture and the next. This could result in two gestures being captured as one, or the operator's normal finger movements being captured as a gesture, resulting in false detection.

For example, if a gesture of moving a finger horizontally involves the finger coming to a near standstill midway and then moving further in the same direction, the horizontal movement gesture may be detected once, or it may be determined that it was repeated twice in succession. Also, if the operator realizes their mistake midway through the movement and moves in the opposite direction, only one of the movements may be valid, or both may be valid. In this way, there is a possibility that a gesture made by the operator may be detected as a gesture different from the operator's intention.

Furthermore, conventional operation input systems have the limitation that finger gestures cannot be detected correctly unless the palm or arm is in a specific position (for example, the palm is parallel to the ground). Addressing these issues requires the use of multiple types of sensors and extremely complex calculations.

In contrast, in the operation input system 2 according to this embodiment, the operator first performs a first gesture using the thumb F1 and index finger F2, and then performs a second gesture following the first gesture. When the control device 30 detects a combination of the first gesture and the subsequent second gesture, it outputs a command that is pre-associated with the second gesture.

Therefore, the operation input system 2 according to this embodiment has the advantage of having high accuracy in detecting and determining gestures and low computational load. Furthermore, since the operation input system 2 is configured to detect the second gesture based on the posture during the first gesture, the operator can input gestures in any stationary posture, which increases the degree of freedom during operation.

[Third embodiment]
Hereinafter, an operation input system 3 according to the third embodiment will be described. The following mainly describes the differences from the first and second embodiments, and descriptions of common parts will be omitted or simplified.

FIG. 21 is a block diagram showing an example of the hardware configuration of the devices constituting the operation input system 3 according to this embodiment. As shown in FIG. 21, the smart glasses 40 include an MCU 401, a wireless communication device 402, a battery 403, and a display device 404. The operation input system 3 according to this embodiment differs from the first and second embodiments in that it further includes smart glasses 40.

The smart glasses 40 are a wearable display that can be worn by the operator as glasses. The smart glasses 40 have, for example, an AR (Augmented Reality) function, a camera function, a Bluetooth/Wi-Fi connection function, a microphone function, etc.

Figure 22 is a perspective view showing an example of smart glasses 40 according to this embodiment. Here, it is shown that the smart glasses 40 include a spectacle frame 41 and a display device (display unit) 404 that projects a screen onto lenses 42 fitted into the spectacle frame 41. An operation input screen output from the control device 30, such as that shown in Figure 20, is projected onto the display device 404.

The smart glasses 40 according to this embodiment are controlled by the control device 30, for example, in the following manner. In a mode in which only images and videos are displayed, the control device 30 disables operation input by gestures through the OS or an application. In addition, when the control device 30 switches the execution mode to a mode in which input by an operator is required, the control device 30 enables operation input by gestures through the OS or an application.

This allows a combination of the first gesture and the subsequent second gesture to be detected, making it possible to input operations using gestures. For example, the operator inputs operations using an operation input screen such as that shown in FIG. 20. If the execution mode is then switched to a mode that only displays images and videos, the OS or application will again disable operation inputs using gestures.

According to the operation input system 3 of this embodiment, an operator wears a ring-shaped device on the thumb F1 and index finger F2 and performs a predetermined gesture to perform operation input. An operation input screen is displayed on the smart glasses 40. The operation input screen is updated in real time according to operation input by the operator's finger gestures.

For example, if the control device 30 is a smartphone, the operator can perform the desired operation input while looking at the screen displayed on the smart glasses 40, not the screen displayed on the smartphone. Therefore, the operator can check the contents of the operation input performed based on the gestures of the first sensor device 10 and the second sensor device 20 through the smart glasses 40, enabling efficient operation input.

[Modified embodiment]
The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit of the present invention.

For example, the first sensor device 10 and the second sensor device 20 may have an inertial measurement unit as a sensor module that further includes sensors such as an acceleration sensor and a magnetic sensor in addition to a gyro sensor. Also, the first sensor device 10 and the second sensor device 20 may use other sensors such as an acceleration sensor instead of a gyro sensor as a sensor module.

In addition, instead of the peak value of the angular velocity in the extraction period, the integrated value of the angular velocity in a predetermined period may be used. By identifying the change pattern of the integrated value, it is possible to identify what kind of finger movement was performed. This also applies to the case where the first sensor device 10 and the second sensor device 20 are configured to detect acceleration instead of angular velocity.

The setting information for the first sensor device 10 and the second sensor device 20 is set to match the finger on which they are to be worn. For this reason, it is preferable that the outer periphery of the base 11 of each of the first sensor device 10 and the second sensor device 20 has characters or figures indicating the finger on which they are to be worn and the direction in which they are to be worn. This allows the operator to wear the first sensor device 10 and the second sensor device 20 in an optimal state on each finger by referring to the characters or figures on the outer periphery of the first sensor device 10 and the second sensor device 20, thereby further improving the accuracy of finger movement detection.

In the above-described embodiment, the finger movement was determined based on the relationship between the first and second angular velocities detected by the first and second sensor devices 10 and 20 and a predetermined threshold value. However, the method of determining the finger movement is not limited to this. For example, the control device may be configured to determine the finger movement by inputting the detected first and second signals to a learning model that has learned finger movements corresponding to the first angular velocity (first signal) and the second angular velocity (second signal). When the learning model is used to determine the finger movement, there is an effect of further reducing the calculation load.

In the above-described embodiment, the description is based on the premise that the posture of the hand is almost maintained during the period from when the operator makes the first gesture to when the operator completes the second gesture. However, if the posture of the operator's hand is significantly different between when the operator makes the first gesture and when the operator makes the second gesture, the accuracy of gesture detection may decrease or the calculation load may increase. Therefore, it is preferable that the control device 30 further includes a configuration for estimating the posture of the operator's hand other than the first finger and the second finger based on the first angular velocity (first signal) and the second angular velocity (second signal). Specifically, the direction of the palm and the back of the hand is estimated. By accurately estimating the posture of the hand, the accuracy of detection of the first gesture and the second gesture can be improved.

The control device 30 may also determine the second gesture on the condition that the change in hand posture after the detection of the first gesture is within a predetermined range. This can suppress a decrease in the accuracy of gesture detection. It also has the effect of reducing the calculation load on the control device 30. Furthermore, if the control device 30 determines that the estimated hand posture during the second gesture is significantly different from the hand posture during the first gesture, it is preferable that the control device 30 can provide feedback to that effect to the operator.

In the above-described embodiment, the first gesture and the second gesture are detected based on the peak value of the angular velocity in the extraction period and a common threshold value. However, the threshold value used to detect the first gesture may be set to a value different from the threshold value used to detect the second gesture. For example, the absolute value of the threshold value when detecting the second gesture may be smaller than the absolute value of the threshold value when detecting the first gesture. In this case, the absolute value of the angular velocity of the thumb F1 and the index finger F2 in the second gesture is smaller than the absolute value of the angular velocity of each finger in the first gesture. In addition, since the sensitivity when detecting the second gesture is higher than when detecting the first gesture, the operator can perform the second gesture after detecting the first gesture with a smaller movement than when performing the first gesture.

Reference Signs List 1, 2, 3... Operation input system 10... First sensor device 20... Second sensor device 30... Control device 30A... Calibration unit 30B... Angular velocity calculation unit 30C... Peak value extraction unit 30D... Command issuing unit 30E... Feedback control unit 30F... Display control unit 30G... Input unit 30H... Gesture detection unit 40... Smart glasses 101... MCU
102: Sensor module 103: Wireless communication device 104: Feedback module 105A: Tightening mechanism 105B: Vibration mechanism 105: Battery 201: MCU
202: Sensor module 203: Wireless communication device 204: Feedback module 205: Battery 301: Processor 302: RAM
303...ROM
304: Storage 305: Communication I/F
306: Display device 307: Input device 401: MCU
402: Wireless communication device 403: Display device 404: Battery

Claims (33)

a first sensor device having an annular base body that can be worn on a first finger of an operator and that outputs a first signal corresponding to a movement of the first finger;
a second sensor device having an annular base body that can be worn on a second finger different from the first finger and that outputs a second signal corresponding to a movement of the second finger;
a control device that determines a relative finger motion of the first finger and the second finger based on the first signal and the second signal;
An operation input system comprising: the first sensor device includes a first wireless communication device configured to wirelessly transmit the first signal to the control device;
the second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication;
The operation input system according to claim 1 . Each of the first sensor device and the second sensor device includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor.
The operation input system according to claim 1 . Each of the first sensor device and the second sensor device includes a plurality of sensor modules.
The operation input system according to claim 3 . The plurality of sensor modules are arranged at substantially equal intervals on the circumference of the base body.
The operation input system according to claim 4 . The outer periphery of the base body is provided with letters or figures indicating the finger on which the device is to be worn and the direction in which the device is to be worn.
The operation input system according to claim 1 . the control device determines the mounting states of the first sensor device and the second sensor device based on the first signal and the second signal.
The operation input system according to claim 1 . the control device performs calibration of the first signal and the second signal at the start-up of the first sensor device and the second sensor device.
The operation input system according to claim 1 . Each of the first sensor device and the second sensor device includes a feedback module that performs a notification operation to the operator.
The operation input system according to claim 1 . The feedback module has a clamping mechanism for clamping the first finger or the second finger.
The operation input system according to claim 9 . The tightening mechanism includes an inflatable or deflated air bag.
The operation input system according to claim 10. The fastening mechanism includes an expandable or contractible dielectric member.
The operation input system according to claim 10. The feedback module has a vibration mechanism that applies vibration to the first finger or the second finger.
The operation input system according to claim 9 . The vibration mechanism has a piezoelectric element that can expand or contract.
The operation input system according to claim 13. The vibration mechanism has an electromagnetic drive unit capable of vibrating.
The operation input system according to claim 13. a first sensor device that is attached to a first finger of an operator and detects a first signal corresponding to a movement of the first finger;
a second sensor device attached to a second finger different from the first finger and configured to detect a second signal corresponding to a movement of the second finger;
A control device that controls each of the first sensor device and the second sensor device;
Equipped with
The control device includes:
Detecting a relative finger motion of the first finger and the second finger based on the first signal and the second signal;
on a condition that the finger motion is determined to be a predetermined first gesture, determining whether or not the finger motion performed following the first gesture is a predetermined second gesture;
When the finger motion is determined to be the second gesture, a command corresponding to the second gesture is output.
Operation input system. the control device starts determining the finger motion when the first signal or the second signal exceeds a threshold.
The operation input system according to claim 16. The threshold value for determining the first gesture is greater than the threshold value for determining the second gesture.
The operation input system according to claim 17. the control device estimates a posture of a hand of the operator other than the first finger and the second finger based on the first signal and the second signal.
The operation input system according to claim 16. the control device determines the second gesture on the condition that a change in the posture of the hand after the detection of the first gesture is within a predetermined range.
The operation input system according to claim 19. the control device determines the finger motion by inputting the detected first signal and the second signal into a learning model that has learned the finger motion corresponding to the first signal and the second signal.
The operation input system according to claim 16. The first finger is a thumb and the second finger is an index finger,
The first gesture is an action of moving the thumb and the index finger apart and then touching the thumb to a side surface of the index finger.
The operation input system according to claim 16. The second gesture is an action of moving the thumb leftward, rightward, forward, and backward along the side surface of the index finger while the thumb and the index finger are in contact with each other.
The operation input system according to claim 22. The second gesture is an action of touching the thumb and the index finger, pressing down the thumb, and touching the thumb to the index finger and the middle finger.
The operation input system according to claim 22. the second gesture is an action of moving the thumb and the index finger apart from each other;
The operation input system according to claim 22. The second gesture is an action of touching the thumb to the index finger after continuously shaking the thumb up and down a predetermined number of times.
The operation input system according to claim 22. the control device outputs the command to move a position of a pointer or a cursor displayed on an operation input screen to any one of a leftward direction, a rightward direction, an upward direction, and a downward direction in response to the second gesture.
The operation input system according to claim 22. the control device outputs, in response to the second gesture, the command for canceling an operation input performed immediately before on an operation input screen.
The operation input system according to claim 22. the control device outputs, in response to the second gesture, the command to reset a flag indicating a detection state of the first gesture.
The operation input system according to claim 22. the control device outputs, in response to the second gesture, the command for determining a previously performed operation input on an operation input screen.
The operation input system according to claim 22. the first sensor device includes a first wireless communication device configured to wirelessly transmit the first signal to the control device;
the second sensor device includes a second wireless communication device that transmits the second signal to the control device by wireless communication;
The operation input system according to claim 22. Each of the first sensor device and the second sensor device includes at least one of a gyro sensor, an acceleration sensor, and a magnetic sensor.
The operation input system according to claim 22. a wearable display including a spectacle frame and a display unit that projects an operation input screen output from the control device onto lenses fitted in the spectacle frame;
The operation input system according to claim 22, further comprising:

Images