JP4447823B2 - Portable information equipment - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a portable information device that provides a predetermined information processing service in response to an input operation from a user, and in particular, operates in response to a user input via an input device that is provided as a standard in the device. It relates to portable information equipment.
[0002]
More particularly, the present invention relates to a portable information device that realizes a complicated input operation with a relatively simple user operation, and more particularly, an input operation including a physical user interface that accepts a user's physical gesture. The present invention relates to a portable information device.
[0003]
[Prior art]
The human hand is an excellent tool. That is, various complicated operations can be accomplished by effectively controlling the many degrees of freedom of the hands and fingers. For example, a musician such as a violinist can play a string with a bow simultaneously in two directions (eg, in the same direction as the string and across it), or simultaneously apply different tensions to the string.
[0004]
Similarly, in modern computers, the position and force can be input simultaneously on the computer screen using an input device such as a mouse or pen. For example, a button is pressed while pointing a specific place on the screen with a mouse, or a pen is pressed on a tablet.
[0005]
In the paper “Human performance evaluation of manipulation schemes in virtual environments” (Proceedings of VRAIS'93. 1993. IEEE. Pp. 155-61) co-authored by S. Zhai and P. Milgram, position control (isotonic control) and Force control (isometric control) suggests that the physiological and psychological mechanisms of manual control are different and that the control direction is basically orthogonal for humans.
[0006]
In portable or handheld devices, it has been a long and important issue that data input is a difficult task and effective interaction cannot be performed. Usually, the input function in a portable device is limited to, for example, pen input via a touch screen, buttons, a jog dial type controller, and the like. In addition, when a touch screen is used, the display content on the screen may be interrupted, the use of a pen is often required, and pen input may be difficult due to the limited resolution of the touch sensor.
[0007]
Interaction via the touch screen is recommended for certain interactions, such as performing direct manipulations on GUI interface objects directly. For example, in order to zoom the map, scrolling and zooming functions must be applied continuously. If such an input operation can be realized with a single gesture, the operation is remarkably simplified and the burden on the user can be reduced.
[0008]
Some proposals have been made regarding a physical user interface in which a user performs all computer operations by applying a physical gesture to a portable computer and realizes various tasks.
[0009]
For example, US Pat. No. 6,243,074 includes a feedback module that displays information about a processed data structure and a detector that detects a user's manual operation, and the data structure in response to the manual operation. A portable device that changes the display mode is disclosed. In addition, US Pat. No. 5,602,566 discloses a small information processing apparatus that can be held with one hand and scrolls the screen according to the inclination of the apparatus body. In these cases, the interface is controlled by detecting an action applied by the user to the device by a sensor built in the device body.
[0010]
Also, J. Rekimoto's paper “Tilting operations for small screen interfaces” (Proceedings of UIST'96. 1996. ACM. Pp. 167-168) uses tilt sensors to detect the tilt of equipment. Is used for scrolling in information display.
[0011]
However, these interfaces focus on realizing certain functions. In other words, the physical interaction with the computer is considered to be asynchronous, eg, user actions follow after forming other actions. Until now, no device that can handle multiple gestures in a transparent combination has been developed.
[0012]
For example, a paper by R. Balakrishnan, G. Fitzmaurice, G. Kurtenbach, and K. Singh “Exploring interactive curve and surface manipulation using a bend and twist sensitive input strip” (Proceedings of Symposium on Interactive 3D graphics. 1999. ACM. Pp. 111-118) and US Pat. No. 5,396,265 disclose a flexible interface that uses a rotation sensor to deflect sensing components that are mechanically connected to each other. . However, such an interface merely focuses on the task of generating a shape, and does not propose a general-purpose interface to be applied to a portable device or even a general device interface.
[0013]
For example, physical interaction for desktop computers has been realized using force sensing devices such as space balls. However, the use of this type of device is generally limited to navigation in a three-dimensional space.
[0014]
[Problems to be solved by the invention]
An object of the present invention is to provide an excellent portable information device capable of realizing a complicated input operation with a relatively simple user operation.
[0015]
It is a further object of the present invention to provide an excellent portable information device that has a physical user interface that accepts a user's physical gesture and can simplify input operations.
[0016]
A further object of the present invention is to provide an excellent portable information device that can transparently combine a plurality of gestures performed by a user and handle it as an input to the device.
[0017]
It is a further object of the present invention to provide an excellent portable information device that can use physical interaction with a device as a general-purpose interface to the device.
[0018]
[Means and Actions for Solving the Problems]
The present invention has been made in view of the above problems, and a first aspect thereof is a portable information device that operates in response to a user input,
The device body,
Gesture input means for acquiring physical interaction applied by a user to the device body;
Processing means for executing processing according to user input;
It is a portable information device characterized by comprising.
[0019]
For example, by collecting gestures such as bending or bending the device body as user input, the user can intuitively operate the device according to his / her feelings and impulses, so that efficient physical interaction can be realized. it can.
[0020]
The portable information device according to the present invention is disposed on the front surface of the device main body and visually displays the processing result of the processing means, and is disposed on the back surface of the device main body, and the user's fingertip. There may be further provided a direction input means for performing a direction input or a coordinate instruction input on the display screen in accordance with the operation by.
[0021]
The portable information device according to the present invention may further include a force sense presentation unit that forcefully feeds back a processing result obtained by the processing means to the user. In such a case, the degree and number of physical interactions that are effectively applied to the device body via the gesture input means can be fed back to the user.
[0022]
The gesture input means includes, for example, an operation unit rotatably coupled to both left and right ends of the device main body, a rotation sensor that detects a rotation operation amount of the operation unit with respect to the device main body, and a sensor output of the rotation sensor And a data acquisition unit that supplies the processing means as a gesture input.
[0023]
In such a case, the user can bend the operation unit with respect to the device body by holding the left and right operation units with their respective hands. At this time, the rotation operation amount detected by the rotation sensor is acquired as a gesture input.
[0024]
The gesture of bending the operation unit with respect to the device main body can be performed intuitively according to the user's emotions and impulses, and is an efficient physical interaction.
[0025]
Such gesture input can be performed while confirming the display content of the visual display. In addition, since the user performs gesture input by grasping the left and right edges of the device body, there is no occlusion problem that the display screen is blocked.
[0026]
Also, the user can give a two-dimensional coordinate instruction by scanning the finger on the back of the device while confirming the display content on the display screen in front of the device. In such a case, the fingertip that performs the coordinate instruction does not block the display screen. In addition, a GUI operation screen is prepared on the front visual display, and a cursor is displayed on the GUI operation screen in accordance with the coordinate instruction position on the touch sensor on the rear side. The two-dimensional coordinate instruction input and the GUI operation on the visual display may be linked.
[0027]
Alternatively, by making the device main body a flexible structure, the gesture input means gestures a deflection sensor for detecting a deflection amount of the device main body due to physical interaction by a user, and a sensor output of the deflection sensor. A data acquisition unit that supplies the processing means as an input can be used. The deflection sensor can detect a deflection amount and a deflection direction of the device main body, and map data and commands on a computer to a combination of the deflection amount and the deflection direction.
[0028]
In such a case, since the device main body is deformed, a flexible display device such as an organic EL element is used for the visual display, and the flexible capacitive mesh sensor is used as a touch sensor. It should be used for.
[0029]
Alternatively, the gesture input means gestures an operation unit attached to both right and left ends of the device body, a force sensor that detects an external force that deforms the operation unit with respect to the device body, and a sensor output of the rotation sensor. A data acquisition unit that supplies the processing means as an input can be used. In this case, when the user applies an external force that bends the device main body to the operation unit, the force sensor can detect this as a gesture input.
[0030]
Alternatively, the gesture input unit includes a pressure sensor that detects an external force applied to deform the device main body, and a data acquisition unit that supplies the sensor output of the pressure sensor to the processing unit as a gesture input. You can also.
[0031]
For example, when the user grips an end of the device main body so as to be sandwiched between a thumb and an index finger and applies an external force to bend it, the pressure sensor detects the pressure applied from the thumb as a gesture input.
[0032]
Also, by arranging force sensors on the upper and lower surfaces of the device main body, not only simply detecting the strength of the force, but also the direction of the physical interaction applied by the user, that is, the device main body in either the upper or lower direction It is possible to determine whether or not it is bent.
[0033]
The processing unit may transparently process a gesture input via the gesture input unit and a direction input via the direction input unit simultaneously.
[0034]
In such a case, the processing means applies a process corresponding to a physical interaction via the gesture input means to an object on the display screen designated via the direction input means. Can do.
[0035]
For example, while scrolling the display location of the map based on the designated coordinates via the direction input means, the scale is switched according to the input gesture (zoom is performed as the physical operation amount increases). Interaction can be performed.
[0036]
The gesture input means includes a force sensor that detects the strength of physical interaction applied by the user to the device main body, and the processing means outputs a continuous sensor output from the force sensor for interface control. It may be handled as an analog value. Then, how much analog value is input via the gesture means may be fed back to the user by force sense presentation unit.
[0037]
Alternatively, in response to the sensor output from the force sensor exceeding a predetermined threshold, the processing means issues a command to the system, switches the interface state and operation mode, and executes other GUI operations. For example, the system operation may be controlled. Further, the user may be happily notified that the physical interaction via the gesture means has been effectively performed.
[0038]
For example, the user may select a desired element on the menu via the direction input means and perform a physical interaction that causes the device body to bend. Then, in response to the external force applied at this time reaching a predetermined threshold value, the corresponding menu command is executed.
[0039]
Alternatively, the processing means may recognize a force pattern detected by the force sensor and handle it as a specific command.
[0040]
For example, various gestures can be created by bending the device body in the same direction or in the opposite direction, bending at different times, or bending at a predetermined interval. Then, commands for system operation can be assigned to these individual gesture patterns.
[0041]
In addition, the user may be happily notified that the physical interaction has been recognized through the gesture means and that the corresponding command has been successfully executed.
[0042]
In short, the present invention provides a computer interface in which the computer interface can perform direction input or coordinate indication and force transmission at the same time. Since multi-dimensional user input can be performed at the same time, the operability is enhanced and the expressiveness of the interaction is increased.
[0043]
Using such a combination of user inputs is particularly effective in handheld, portable, and portable computers, and can provide rich interaction to the user.
[0044]
According to such an interface, the user can realize interaction related to different orthogonality information by only a single gesture. For example, an interaction such as scrolling a city map and simultaneously changing the scale can be performed with a single gesture.
[0045]
Other objects, features, and advantages of the present invention will become apparent from more detailed description based on embodiments of the present invention described later and the accompanying drawings.
[0046]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0047]
FIG. 1 schematically shows a hardware configuration of a portable device 100 according to an embodiment of the present invention.
[0048]
As shown in the figure, the portable device 100 includes a processor unit 108 as a main controller, a force sensor 101, a deflection sensor 102, a pressure sensor 103, a rotation sensor 104, and two users as interfaces for performing user input. A touch sensor 105 for inputting a dimensional position, a visual display 106 as an interface for performing user output, and a tactile display 109 are provided.
[0049]
The processor unit 108 is, for example, a CPU (Central Processing Unit) as a processing engine, a RAM (Random Access Memory) for loading an execution program or temporarily storing work data, a program code, and other data permanently. An application using physical interaction such as a user gesture is executed in an execution environment provided by an operating system (OS), which is configured by a combination of ROM (Read Only Memory) to be stored.
[0050]
The visual display 106 is composed of a liquid crystal display (LCD), for example, and visually displays the processing result in the processor unit 108.
[0051]
The touch sensor 105 may be superimposed on the visual display 106, for example. Alternatively, the device 100 is disposed on the opposite side of the main body of the device 100 from the visual display 106 (that is, the back side of the device 100). In the latter case, the user can give a two-dimensional coordinate instruction by scanning the back of the device 100 with a fingertip while confirming the display content on the display screen in front of the device 100. Therefore, the fingertip that performs the coordinate instruction does not block the display screen of the visual display 106.
[0052]
The haptic display 109 is a device for hapticly feeding back the processing result in the processor unit 108. By applying an arbitrary control signal to the haptic display 109, the haptic pattern can be freely changed. The processor unit 108 may generate a control signal that is a time function of voltage according to the processing result data. For example, the haptic display 109 can be constructed with a multilayer piezo flexible actuator. A multilayer piezo actuator consists of an upper layer piezo actuator and a lower layer piezo actuator. By applying reverse voltages to the upper and lower piezo actuators, the upper layer contracts and the lower layer expands simultaneously. Thus, the entire multilayer piezoelectric flexible actuator can be bent upward or downward. The force display is described in Japanese Patent Application No. 2002-18228 already assigned to the present applicant.
[0053]
The force sensor 101, the deflection sensor 102, the pressure sensor 103, and the rotation sensor 104 are devices for detecting physical interactions such as user gestures and capturing them as computer data. In order to acquire a physical interaction, the portable device 100 does not need to be equipped with all of these sensors 101 to 104, and may be provided with at least one of them.
[0054]
The data acquisition unit 107 digitizes these sensor outputs and supplies them to the processor unit 108. In the processor unit 108, the two-dimensional position input from the touch sensor 105 and the gesture input via the data acquisition unit 107 are simultaneously and transparently processed, so that different orthogonality information is related. Interaction can be achieved with only a single gesture. The processor unit 108 provides, for example, an interaction in which the scale is simultaneously changed while scrolling a city map. Gesture-based interaction is intuitive, efficient, and fun.
[0055]
FIG. 2 shows an example of the external configuration of the portable device 100. In the example shown in the figure, the operating unit 111 is hingedly coupled to both left and right edges of the portable device 100 main body. In addition, a rotation sensor 104 is provided at the hinge portion, and the amount of rotation operation of the operation unit 111 with respect to the device 100 main body can be detected.
[0056]
For example, the user can bend the operation unit 111 with respect to the main body of the device 100 by holding the left and right operation units 111 with respective hands as illustrated. At this time, the rotation operation amount detected by the rotation sensor 104 is given to the processor unit 108 via the data acquisition unit 107 as a gesture input. Such gesture input can be performed while confirming the display content of the visual display 106. Since the user performs gesture input by grasping the left and right side edges of the device 100 main body, there is no occlusion problem that the display screen is blocked.
[0057]
The gesture of bending the operation unit 111 with respect to the main body of the device 100 can be performed intuitively according to the user's emotions and impulses, and is an efficient physical interaction.
[0058]
FIG. 3 shows a modification of the portable device 100 shown in FIG.
[0059]
As shown in the drawing, a visual display 106 is disposed on the front side of the main body of the device 100, and a touch sensor 105 is disposed on the opposite side, that is, on the back side of the device 100. Accordingly, the user can give a two-dimensional coordinate instruction by scanning the back surface of the device 100 with the fingertip while confirming the display content on the display screen in front of the device 100. In such a case, the fingertip that performs the coordinate instruction does not block the display screen. In addition, a GUI operation screen is prepared on the visual display 106 on the front side, and a cursor is displayed on the GUI operation screen in accordance with the coordinate instruction position on the touch sensor 105 on the back side, so that the touch sensor is displayed. A two-dimensional coordinate instruction input on 105 and a GUI operation on the visual display 106 may be linked.
[0060]
The processor unit 108 can simultaneously and transparently process a gesture input using the left and right operation units 111 and a two-dimensional coordinate instruction input via the touch sensor 105. For example, in a map display application in an urban area, the scale is switched in accordance with the gesture, that is, the rotation operation amount of the operation unit 111 while scrolling the map display location based on the coordinates indicated by the touch sensor 105 (rotation operation amount Zoom in as you increase).
[0061]
FIG. 4 shows still another modification of the portable device 100.
[0062]
As shown in the drawing, a visual display 106 is disposed on the front side of the main body of the device 100, and a touch sensor 105 is disposed on the opposite side, that is, on the back side of the device 100. The user can give a two-dimensional coordinate instruction by scanning the touch sensor 105 on the back of the device 100 with a fingertip while confirming the display content on the display screen in front of the device 100. In such a case, the fingertip that performs the coordinate instruction does not block the display screen.
[0063]
Of course, a GUI operation screen is prepared on the visual display 106 on the front side, and a cursor is displayed on the GUI operation screen in accordance with the coordinate instruction position on the touch sensor 105 on the back side, so that the touch sensor is displayed. A two-dimensional coordinate instruction input on 105 and a GUI operation on the visual display 106 may be linked.
[0064]
Further, in this modification, as a gesture input means, not the amount of rotation of the operation unit with respect to the main body of the device 100 but a deflection applied to the main body of the device 100 is used. In this case, the physical interaction of the user is measured based on the sensor output from the deflection sensor 102 built in the device 100 main body. Then, the processor unit 108 acquires a gesture input via the data acquisition unit 107.
[0065]
In this embodiment, the device 100 main body is deformed in response to a gesture input. For this reason, a flexible display device such as an organic EL element is used for the visual display 106, and a flexible capacitive mesh sensor may be used for the touch sensor 105. .
[0066]
FIG. 5 shows still another modification of the portable device 100.
[0067]
As shown in the drawing, a visual display 106 is disposed on the front side of the main body of the device 100, and a touch sensor 105 is disposed on the opposite side, that is, on the back side of the device 100. The user can perform a two-dimensional coordinate instruction by scanning the touch sensor 105 on the back surface of the device 100 while checking the display content on the display screen in front of the device 100. The fingertip does not block the display screen.
[0068]
Of course, a GUI operation screen is prepared on the visual display 106 on the front side, and a cursor is displayed on the GUI operation screen in accordance with the coordinate instruction position on the touch sensor 105 on the back side, so that the touch sensor is displayed. A two-dimensional coordinate instruction input on 105 and a GUI operation on the visual display 106 may be linked.
[0069]
Further, in the example shown in the figure, the operation unit 111 is connected to the left and right end edges of the portable device 100 via the force sensor 101, respectively. Therefore, when the user applies an external force that causes the device 100 to bend to the operation unit 111, the force sensor 101 detects this as a gesture input.
[0070]
Therefore, the user can perform transparent gesture input simultaneously with coordinate instruction input by holding the left and right sides of the apparatus 100 with both hands and pressing the thumb while looking at the visual display 106. The gesture of bending the main body of the device 100 can be performed intuitively according to the user's emotions and impulses, and is an efficient physical interaction.
[0071]
In this embodiment, since the main body of the device 100 is not deformed in response to the gesture input, it is not always necessary to use a flexible device as the visual display 106.
[0072]
FIG. 6 shows still another modification of the portable device 100.
[0073]
As shown in the drawing, a visual display 106 is disposed on the front side of the main body of the device 100, and a touch sensor 105 is disposed on the opposite side, that is, on the back side of the device 100. The user can give a two-dimensional coordinate instruction by scanning the touch sensor 105 on the back of the device 100 with a fingertip while confirming the display content on the display screen in front of the device 100. In such a case, the fingertip that performs the coordinate instruction does not block the display screen.
[0074]
Of course, a GUI operation screen is prepared on the visual display 106 on the front side, and a cursor is displayed on the GUI operation screen in accordance with the coordinate instruction position on the touch sensor 105 on the back side, so that the touch sensor is displayed. A two-dimensional coordinate instruction input on 105 and a GUI operation on the visual display 106 may be linked.
[0075]
Further, in this modification, pressure sensors 103 are disposed on both the front and back sides of the apparatus 100 main body as the gesture input means. Therefore, when the user grips the end of the main body of the apparatus 100 with the thumb and index finger or the like and applies an external force to bend it, the pressure sensor 103 detects the pressure applied from the thumb as a gesture input.
[0076]
Therefore, the user applies a force to bend upward so that the surface of the device 100 body sticks out while holding the left and right sides of the device 100 body with both hands and looking at the visual display 106 with a posture sandwiched between a thumb and an index finger. Or by applying a downward force so as to dent the surface, a transparent gesture input can be performed simultaneously with a coordinate instruction input. The gesture of bending the main body of the device 100 can be performed intuitively according to the user's emotions and impulses, and is an efficient physical interaction.
[0077]
In this embodiment, since the main body of the device 100 is not deformed in response to the gesture input, it is not always necessary to use a flexible device as the visual display 106.
[0078]
A mechanism for realizing gesture input using the pressure sensor 103 will be described with reference to FIGS.
[0079]
As shown in FIG. 7, two pressure sensors 103 are attached to the front surface and the back surface of the main body of the device 100, respectively. The pressure sensor 103 is touched on each surface using the thumb and index finger of one hand.
[0080]
In this way, the user picks it so that it is sandwiched between the thumb and forefinger, and applies a force that deflects upward so as to project the surface of the main body of the device 100 or a force that applies downward force that dents the surface. can do. In which direction the user is applying force to the main body of the device 100 can be detected by the difference in sensor output between the pressure sensors 103 arranged on the front and back surfaces. That is, if the output of the pressure sensor 103 on the front side is larger than that on the back side, a downward force is applied and it is determined that a downward gesture input is being performed (see FIG. 8). .
[0081]
Of course, the pressure sensor 103 may be disposed only on either the front or the back to simplify the structure. In such a case, the force can be applied only in either one, and the gesture input is performed only with the magnitude of the pressure.
[0082]
The present invention provides a computer interface in which the computer interface can simultaneously perform coordinate indication and force transmission. As shown in FIGS. 3 to 6, as an essential configuration of the portable device 100 according to the present invention, the touch panel 105 is disposed on the back surface of the device 100 main body. With such a configuration, the user can naturally hold the device 100 by gripping both left and right ends of the device body with both hands. Then, the user can perform a gesture input by transmitting force to the main body of the device 100 by hand, and at the same time, by freely moving a fingertip placed on the touch panel 105, a coordinate instruction input can be performed transparently. it can.
[0083]
According to such a computer interface, a complicated interaction can be realized for a handheld computer without using a keyboard, a mouse, a pen, or the like. There is also no problem of blocking the visual display 106 of the device. Therefore, a comfortable operating environment can be provided by combining this computer interface with a device having only a small display.
[0084]
The portable device 100 has a form factor that makes it easy for a user to perform a gesture input interaction by applying a force to the device body while placing a fingertip on the touch panel 105 and performing an interaction for coordinate instruction. It is preferable.
[0085]
According to the portable device 100 according to the present embodiment, the coordinate information is input to the display information or the GUI object provided by the visual display 106, and at the same time, the specific position indicated in response to the gesture input is performed. Further processing can be applied to it. Hereinafter, a method for simultaneous input of such data in the portable device 100 according to the present embodiment will be described.
[0086]
(1) Combination of coordinate command input and continuous force control:
In this case, the force applied by the user to the main body of the device 100 is mapped to an analog value in the interface control.
[0087]
For example, a map is displayed on the visual display 106, the location on the map is indicated using the touch panel 105 on the back of the device 100, and the physical interaction of the user is detected by the force sensor 101. The scale of the map is controlled according to this sensor output. If the amount of deflection applied to the device 100 is small, the map display on the visual display 106 is enlarged or reduced. If the device 100 is a rigid body and does not actually deform even when a force is applied, how much analog value is input to the device 100 main body is fed back to the user using the haptic display 109. Also good.
[0088]
(2) Combination of coordinate instruction input and force control exceeding threshold:
When the force applied to the main body of the device 100 by the user exceeds a predetermined threshold, a command for the system is issued, the interface state and the operation mode are switched, and other GUI operations are executed.
[0089]
For example, the user can browse the menu by moving the selector on the menu via the touch panel 105. What is necessary is just to perform the physical interaction which bends the apparatus 100 main body further, when a desired element is selected. Then, in response to the external force applied at this time reaching a predetermined threshold value, the corresponding menu command is executed. For example, the haptic display 109 may be used to notify the user that the force of the gesture performed by the user has reached a predetermined level and the physical interaction has been effectively performed.
[0090]
(3) Coordinate command input and gesture-based force control:
A pattern of force applied to the main body of the device 100 is recognized and handled as a specific command.
[0091]
For example, the user scans a desired position on the map displayed on the visual display 106 via the touch panel 105, and further flexes the device 100 body only twice in the vertical direction instantly. Switch the screen display.
[0092]
In such a case, for example, various gestures can be made by bending the main body of the device 100 in the same direction or in the opposite direction, bending at different times, or bending at a predetermined interval. Then, the user may be notified of the recognition of the physical interaction applied to the main body of the device 100 and the successful execution of the corresponding command using, for example, the haptic display 109.
[0093]
FIG. 9 shows a system operation using gesture input in the form of a flowchart. Such processing is actually realized by the processor unit 108 that activates a predetermined program code simultaneously and transparently handling the gesture input and the coordinate instruction input via the touch panel 105.
[0094]
First, the designated coordinate values (X, Y) via the touch panel 105 and the force F detected by the force sensor 101 or the gesture via the deflection sensor 102, the pressure sensor 103, and the rotation sensor 104 are input (step). S1).
[0095]
Here, when the user forms a gesture (for example, when the main body of the device 100 is deformed only twice) (step S2), after a haptic feedback is returned to the user via the haptic display 109 (step S2). S3), a predetermined action such as displaying a menu is executed (step S4). If the user does not make a gesture, reading of the input data is continued (step S5).
[0096]
When a coordinate instruction input is made via the touch panel 105, a GUI object corresponding to the support coordinates is searched (step S6).
[0097]
Next, it is determined whether the designated GUI object is a selectable object such as a menu item or a hyperlink (step S7).
[0098]
If the designated GUI object can be selected, whether the force F applied to the main body of the device 100 by the gesture input exceeds a predetermined threshold (or the degree of the gesture input exceeds a predetermined level). It is determined whether or not (step S8).
[0099]
If the force F exceeds a predetermined threshold, the input gesture is validated and haptic feedback is returned to the user via the haptic display 109 (step S9), and the selected menu is executed. A command based on the designated coordinates, such as searching for a hyperlink or the like, is issued (step S10). If the gesture input is not validated, for example, the force F does not reach a predetermined threshold value, the input data reading is continued (step S11).
[0100]
If it is determined in step S7 that a selectable GUI object is not designated, it is further determined whether or not the GUI object controls continuous values such as a scale, a hue, and a size ( Step S12).
[0101]
When the GUI object controls a continuous value, the force F (or other input gesture) applied to the main body of the device 100 is mapped to a change in control value such as zooming of the map display. (Step S13). And after returning a tactile feedback to a user via the tactile display 109 (step S14), reading of input data is continued (step S15).
[0102]
If the determination result in step S12 is negative, it is further determined whether or not the instructed GUI object has an additional information layer such as help, preview, or another view (step S16).
[0103]
If there is an additional information layer, it is determined whether continuous control by gesture input is possible (step S17). If continuous control by gesture input is possible, the force F (or other input gesture) applied to the main body of the device 100 is used as additional information such as zoom in / out and blending. Mapping to a layer (step S18). And after returning a tactile feedback to a user via the tactile display 109 (step S19), reading of input data is continued (step S20).
[0104]
On the other hand, if continuous control by gesture input is not possible, it is further determined whether or not the force F applied to the main body of the device 100 exceeds a predetermined threshold (or the degree of gesture input exceeds a predetermined level). (Step S21). If the force F exceeds a predetermined threshold, haptic feedback is returned to the user via the haptic display 109 (step S22), and additional information such as pop-up help is displayed (step S22). Step S23). If the gesture input is not validated because the force F does not reach a predetermined threshold value, the input data is continuously read (step S24).
[0105]
If it is determined in step S16 that the instructed GUI object does not have additional information, reading of input data is continued (step S25).
[0106]
As described above, the gesture input according to the present embodiment can be used for interaction in a handheld device that does not use a mouse, a touch screen, a keyboard, or the like. For example, the following uses can be mentioned.
[0107]
(1) Advanced remote control for display screen
(2) Advanced remote control for TV receiver (see Fig. 10)
(3) Viewing and managing PDA (Personal Digital Assistant) and other personal information
(4) Mobile phone
▲ 5 ▼ Electronic book
(6) Handheld game controller
[0108]
FIG. 10 shows a state in which the gesture input according to the present embodiment is applied to a remote controller for audio equipment such as a TV receiver. In the embodiment shown in the figure, the user applies a physical interaction in which the remote controller body is bent in each direction. At this time, the pattern of the amount of bending or the strength of the force applied to the body is changed. In the remote controller, such a gesture is decoded into device control data and commands and transferred to the device body.
[0109]
[Supplement]
The present invention has been described in detail above with reference to specific embodiments. However, it is obvious that those skilled in the art can make modifications and substitutions of the embodiment without departing from the gist of the present invention. That is, the present invention has been disclosed in the form of exemplification, and the contents described in the present specification should not be interpreted in a limited manner. In order to determine the gist of the present invention, the claims section described at the beginning should be considered.
[0110]
【The invention's effect】
As described in detail above, according to the present invention, it is possible to provide an excellent portable information device capable of realizing a complicated input operation with a relatively simple user operation.
[0111]
Further, according to the present invention, it is possible to provide an excellent portable information device that includes a physical user interface that accepts a user's physical gesture and can simplify an input operation.
[0112]
Further, according to the present invention, it is possible to provide an excellent portable information device that can be used as an input to the device by transparently combining a plurality of gestures performed by the user.
[0113]
Furthermore, according to the present invention, it is possible to provide an excellent portable information device that can use physical interaction with a device as a general-purpose interface to the device.
[0114]
ADVANTAGE OF THE INVENTION According to this invention, the computer interface which can perform a coordinate instruction | indication and force transmission simultaneously can be provided. That is, since multi-dimensional user input can be performed simultaneously, operability is enhanced and interaction expressiveness is increased.
[0115]
Using such a combination of user inputs is particularly effective in handheld, portable, and portable computers, and brings rich interaction to users without using a mouse, touch panel, keyboard, etc. Can do. Further, since the interface according to the present invention does not interfere with the display screen, the display content of the screen is not obstructed in the interaction by the gesture.
[0116]
According to the interface according to the present invention, the user can realize the interaction related to the different orthogonality information only by a single gesture. Gesture-based interaction is intuitive, efficient, and fun.
[0117]
According to the interface of the present invention, two-dimensional coordinate instruction and force input can be simultaneously and transparently performed on a small handheld computer. For example, an interaction such as scrolling a city map and simultaneously changing the scale can be performed with a single gesture.
[Brief description of the drawings]
FIG. 1 is a diagram schematically showing a hardware configuration of a portable device 100 according to an embodiment of the present invention.
FIG. 2 is a diagram showing an example of an external configuration of a portable device 100. FIG.
FIG. 3 is a view showing a modification of the portable device 100 shown in FIG. 2;
4 is a diagram showing a modification of the portable device 100 shown in FIG.
FIG. 5 is a view showing a modification of the portable device 100 shown in FIG. 2;
6 is a diagram showing a modification of the portable device 100 shown in FIG. 2. FIG.
FIG. 7 is a diagram for explaining a mechanism for realizing gesture input using a pressure sensor 103;
FIG. 8 is a diagram for explaining a mechanism for realizing gesture input using the pressure sensor 103;
FIG. 9 is a flowchart showing a system operation using a gesture input.
FIG. 10 is a diagram showing an application example of gesture input.
[Explanation of symbols]
100: Portable information device
101: Force sensor, 102: Deflection sensor
103 ... Pressure sensor, 104 ... Rotation sensor
105 ... Touch panel
106: Visual display
107: Data acquisition unit
108: Processor unit
109 ... Haptic display
111 ... operation unit

Claims (11)

A portable information device which operates in response to user input,
The device body,
A first pressure sensor disposed at both left and right ends of the front of the device body;
A second pressure sensor disposed on each of the left and right ends of the back of the device body;
Based on the difference between the outputs of the first and second pressure sensors, a user who picks up the left and right ends of the device main body with fingers and deflects the surface of the device main body upward or the A gesture input means for detecting which of the downward force to dent the surface is applied and determining the direction of the gesture input performed by the user;
Processing means for executing processing according to the gesture input from the user;
A portable information device comprising: A visual display disposed on the front surface of the device body for visually displaying a processing result by the processing means;
A direction input means disposed on the back of the device body for performing direction input on a display screen in accordance with an operation by a user's fingertip;
The portable information device according to claim 1, further comprising: A force sense presentation unit for forcefully feeding back a processing result by the processing means to the user;
The portable information device according to claim 1. A visual display disposed on the front surface of the device body for visually displaying a processing result by the processing means;
A direction input means for inputting a direction on the display screen;
It said processing means processes the direction input through the direction input means and gesture input via the gesture input means simultaneously,
The portable information device according to claim 1. The processing means applies a process according to the gesture input via the gesture input means to an object on the display screen designated via the direction input means.
The portable information device according to claim 4. The processing means handles the continuous detection output detected by the gesture input means as an analog value for interface control.
The portable information device according to claim 1. A haptic presentation unit that happily feeds back to the user how many analog values have been input via the gesture input means;
The portable information device according to claim 6. The processing means controls a system operation in response to a detection output by the gesture input means exceeding a predetermined threshold value.
The portable information device according to claim 1. A haptic presentation unit that happily notifies the user that the gesture input via the gesture input means has effectively acted;
The portable information device according to claim 8. The processing means recognizes the force pattern detected by the gesture input means and handles it as a specific command.
The portable information device according to claim 1. A haptic presentation unit that happily notifies the user that the gesture input has been recognized and that the corresponding command has been successfully executed via the gesture input unit;
The portable information device according to claim 10.

Images