Novest: Position Estimation of Finger on Back of Hand with a Small Ranging Sensor Array

The input method of smartwatches depends on a touch panel and buttons. The size of their interface gets smaller in proportion to the size of the devise; therefore, an operating finger becomes relatively large and hides information on screen, which leads to a problem of lowered operating efficiency.

For this problem, a technique to use the fringe area of the device as an input interface is being suggested, and there are types that use permanent magnet [1], interchange signal [2], and supersonic wave [3].

This article proposes a technique to use BoH like a trackpad with a small ranging sensor array. The two-dimensional position of a finger and distance between the finger and BoH is distinguished by obtaining distance data and signal strength data with a sensor attached to the side of a smart watch. Using BoH as a trackpad allows operation without hiding any information on display. Also, it solves the problem of having a difference between an operating object and an input area on screen, by estimating the two-dimensional position of the finger while distinguishing the presence or absence of distance between the finger and BoH.

This study estimates the position of a finger on BoH, and distinguishes the presence or absence of distance between the operating finger and BoH by attaching a small sensor on the side of a smart watch.

Figure 1 (Left) shows the overview of Novest, and 3 states (out of range, hovering, touching) are differentiated while distinguishing the presence or absence of a distance. Out of range is a state where the sensor does not detect the finger, hovering is a state in which there is a certain level of distance between the finger and BoH, and touching is a state in which there is a small amount of distance between the finger and BoH or they are in contact.

Overview of Novest: (Left) Usage of Novest, (Right) Ranging sensor array of Novest

Full size image

To distinguish the distance between a finger and BoH, ranging sensors are arranged vertically on BoH.

The gap between the operating object on display and the input area on BoH is solved by treating them like the relation between a mouse and a mouse pointer, setting up the touching state. Touching is equivalent to clicking on a mouse.

This article only refers to the case of operating with one finger (index finger on right hand) Fig. 1: (Left).

The implemented ranging sensor array is shown in Fig. 1 (Right). The size of the sensor board is 15 mm in length, 50 mm in width and 8 mm in thickness. The width is determined in reference to the side of SONY Smart Watch 3 [4] (10 mm height, 51 mm width). As ranging sensor, 18 (2 × 9) of Time of Flight system STMicroelectronic VL6180Xs are arranged. This arrangement is made during the preliminary experiment to make sure that a finger 20 mm away from the sensor is in the view area of at least one sensor. The minimal vertical intervals implementable are set, and the intervals in the luminescent part are approximately 8 mm. This sensor board is connected to NXP Semiconductors prototyping board mbed LPC1768 with a flat cable, controlled by I2C communication.

Additionally, the noise of distance data and signal strength data obtained by sensor array is suppressed using a low-path filter.

3.1 Finger Position Estimation

Coordinate system is set as Fig. 1 (Left). Strength data of 9 signals of a top column on the ranging sensor is used for Y coordinate derivation. Since this time one finger is assumed to operate, strength data of 9 signals is processed with spline interpolation, determines the position with the strongest signal as Y coordinate.

After that, the sensor distance data closest to the derived Y coordinate is used as X coordinate.

3.2 Finger State Classification

Support Vector Machine is used to classify the distance between the operating finger and BoH. 44 features are used, and they consist of the signal strength ratio of lower column to upper column (9) and its average value, total value, standard deviation, maximum value, minimum value, range, signal strength difference with the adjacent sensor (16), signal strength difference between the lower column and upper column (9), its average value, standard deviation, total value, and the ratio of total signal strength in lower column to total signal strength in upper column. These are scaled from −1 to 1 for use.

To evaluate the performance of this technique, experiments under 4 situations were carried out, with a finger and a round stick grasped by a robot arm as a direction method, and a flat surface and BoH as a directed surface. Subjects were 9 university and graduate school students (1 female, 8 males, 23.8 years old (SD = 3.7 years)). The purpose is to confirm the finger and BoH effect on this technique by comparing the position estimate accuracy and the distance distinguishing accuracy under 4 experimental situations.

In this evaluation experiment, pointing task which is to point a marker on BoH or a surface with one’s index finger on right hand was implemented, and 20 markers were set up for a random indication.

The task was implemented under the environments of (1) finger and BoH, (2) finger and flat surface, (3) round stick and BoH. The above 3 environments were of evaluation in touching, and after they were completed, same tasks were implemented with hovering in the environment of finger and BoH. With hovering, it was instructed to keep 10–15 mm distance between a finger and BoH.

As a result, the position estimation accuracy of touching was 4.2 mm (SD = 1.2 mm), and the estimate accuracy of hovering was 5.5 mm (SD = 2.8 mm). Additionally, the distance distinguishing accuracy was 91.8% (SD = 8.3%) with per-user classifier.

We proposed Novest which is a technique to make BoH a trackpad by setting up a small ranging sensor array on the side of a smart watch. Novest did not only distinguish the two-dimensional position of a finger, but also the presence or absence of a distance from BoH. Then, accuracies of the position estimate and distinguishing distance between touching and hovering were confirmed by implementing pointing tasks under 4 kinds of environments for experiments. As for the position estimate accuracy, a highly accurate estimate of 4.2 mm (SD = 1.2 mm) was possible with touching. Finally, this technique can be expected to be applied to small devices other than a smart watch such as Smartglasses, because it distinguishes the two-dimensional position of a finger and the presence or absence of distance without attaching a sensor directly on skin.