JP2016103102A - Portable terminal and control method therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To minimize erroneous input by users.SOLUTION: In one embodiment, a portable terminal includes a controller configured to: determine, from a plurality of objects, an object displayed at a first line-of-sight position detected by a line-of-sight detection unit as a target object; determine, when a horizontal movement on a principal surface of a display unit is detected by a movement detection unit, a line-of-sight position detected by the line-of-sight detection unit after the detection of the horizontal movement as a second line-of-sight position; and change a display position of the target object on the basis of the second line-of-sight position when a distance between the first line-of-sight position and the second line-of-sight position is equal to or larger than a threshold.SELECTED DRAWING: Figure 1

Description

  The present application relates to a mobile terminal and a control method.

  Conventionally, in an electronic device including a touch screen, a user may make an input error with respect to an interface displayed on the touch screen. In order to solve such a problem, a technique for improving the accuracy of data input by adjusting the position of a touch input point actually sensed with respect to the touch screen is known (see Patent Document 1). .

Japanese Patent Laid-Open No. 10-91318

  Since the technique disclosed in Patent Literature 1 adjusts the position of the touch input point according to a specific tendency related to user input, particularly when an erroneous input is not based on a specific tendency. When it is caused by vibration, there is a problem that it is difficult to adjust the position of the touch input point.

  From the above, it is necessary to provide a portable terminal and a control method that can prevent a user's erroneous input as much as possible.

  A mobile terminal according to one aspect includes a display unit that displays a plurality of objects for performing an input operation on a main surface, a movement detection unit that detects movement, an image acquisition unit that acquires an image, and the image acquisition unit And a line-of-sight position detection unit that detects a line-of-sight position of a user corresponding to one of the plurality of objects, and a line-of-sight position detection unit that is detected by the line-of-sight position detection unit among the plurality of objects. When an object displayed at one line-of-sight position is determined as a target object and a movement in the horizontal direction with respect to the main surface of the display unit is detected by the movement detection unit, after the movement in the horizontal direction is detected The line-of-sight position detected by the line-of-sight position detection unit is determined as the second line-of-sight position, and the distance between the first line-of-sight position and the second line-of-sight position is within a threshold range. Expediently, a control unit for changing the basis of the display position of the target object in the second line-of-sight position.

  A control method according to one aspect includes a display unit that displays a plurality of objects for performing an input operation on a main surface, a movement detection unit that detects movement, an image acquisition unit that acquires an image, and the image acquisition unit. And a gaze position detection unit that detects a gaze position of a user corresponding to any of the plurality of objects based on the image acquired by Among them, the step of determining the object displayed at the first line-of-sight position detected by the line-of-sight position detection unit as a target object, and the movement in the horizontal direction with respect to the main surface of the display unit are performed by the movement detection unit If detected, the step of determining the line-of-sight position detected by the line-of-sight position detecting unit as the second line-of-sight position after detecting the vibration in the horizontal direction; If the distance between the first viewpoint position and the second line-of-sight position is equal to or higher than the threshold, and a step of changing on the basis of the display position of the target object in the second line-of-sight position.

FIG. 1 is a block diagram illustrating a functional configuration of the smartphone according to the first embodiment. FIG. 2 is a diagram for explaining an example of processing executed by the smartphone 1 according to the first embodiment. FIG. 3 is a diagram for explaining an example of processing executed by the smartphone 1 according to the first embodiment. FIG. 4 is a diagram for explaining an example of processing executed by the smartphone 1 according to the first embodiment. FIG. 5 is a diagram for explaining an example of processing executed by the smartphone 1 according to the first embodiment. FIG. 6 is a diagram for explaining an example of processing executed by the smartphone 1 according to the first embodiment. FIG. 7 is a flowchart illustrating a process flow by the smartphone 1 according to the first embodiment. FIG. 8 is a flowchart showing a flow of processing by the smartphone 1 according to the first embodiment. FIG. 9 is a diagram for explaining an example of processing executed by the smartphone 1 according to the second embodiment. FIG. 10 is a diagram for explaining an example of processing executed by the smartphone 1 according to the second embodiment. FIG. 11 is a diagram for explaining processing of another embodiment.

  A plurality of embodiments for carrying out a portable terminal and a control method according to the present application will be described in detail with reference to the drawings. Hereinafter, a smartphone will be described as an example of the mobile terminal.

(Embodiment 1)
An example of a functional configuration of the smartphone 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a block diagram illustrating a functional configuration of the smartphone according to the first embodiment. In the following description, the same code | symbol may be attached | subjected to the same component. Furthermore, duplicate descriptions may be omitted.

  As shown in FIG. 1, the smartphone 1 includes a touch screen display 2, a button 3, an illuminance sensor 4, a proximity sensor 5, a communication unit 6, a receiver 7, a microphone 8, a storage 9, and a controller 10. A speaker 11, a camera 12, an infrared irradiation unit 13, a connector 14, an acceleration sensor 15, and an orientation sensor 16.

  The touch screen display 2 includes a display 2A and a touch screen 2B. The display 2 </ b> A and the touch screen 2 </ b> B may be arranged, for example, may be arranged side by side, or may be arranged apart from each other. When the display 2A and the touch screen 2B are arranged so as to overlap each other, for example, one or more sides of the display 2A may not be along any side of the touch screen 2B. The touch screen display 2 is an example of a display unit.

  The display 2A includes a liquid crystal display (LCD: Liquid Crystal Display), an organic EL display (OELD: Organic Electro-Luminescence Display), or an inorganic EL display (IELD: Inorganic Electro-Luminescence Display). The display 2A displays characters, images, symbols, graphics, and the like. The screen including characters, images, symbols, graphics, and the like displayed on the display 2A includes a screen called a lock screen, a screen called a home screen, and an application screen displayed during execution of the application. The home screen may be called a desktop, a standby screen, an idle screen, a standard screen, an application list screen, or a launcher screen. The display 2A displays a plurality of objects. Objects include user interface objects (icons) such as applications and objects to be input, such as soft keys.

  The touch screen 2B detects contact of a finger, a pen, a stylus pen, or the like with respect to the touch screen 2B. The touch screen 2B is a position (hereinafter referred to as a contact position) on the touch screen 2B when a plurality of fingers, a pen, a stylus pen or the like (hereinafter simply referred to as “finger”) contacts the touch screen 2B (touch screen display 2). Can be detected. The touch screen 2B notifies the controller 10 of the contact of the finger with the touch screen 2B together with the contact position. The touch screen 2B is an example of a contact position detection unit.

  The detection method of the touch screen 2B may be any method such as a capacitance method, a resistive film method, a surface acoustic wave method (or an ultrasonic method), an infrared method, an electromagnetic induction method, and a load detection method. In the following description, in order to simplify the description, it is assumed that the user uses the finger to touch the touch screen 2B in order to operate the smartphone 1.

  The controller 10 (smart phone 1) has at least one of the contact detected by the touch screen 2B, the position at which the contact is detected, the change in the position at which the contact is detected, the interval at which the contact is detected, and the number of times the contact is detected. The type of gesture is determined based on the one. The gesture is an operation performed on the touch screen 2B (touch screen display 2) using a finger. The gestures that the controller 10 (smart phone 1) determines via the touch screen 2B include, for example, touch, long touch, release, swipe, tap, double tap, long tap, drag, flick, pinch in, and pinch out. However, it is not limited to these.

  The button 3 receives an operation input from the user. The number of buttons 3 may be singular or plural.

  The illuminance sensor 4 detects illuminance. The illuminance is the value of the light beam incident on the unit area of the measurement surface of the illuminance sensor 4. The illuminance sensor 4 is used for adjusting the luminance of the display 2A, for example.

  The proximity sensor 5 detects the presence of a nearby object without contact. The proximity sensor 5 detects the presence of an object based on a change in a magnetic field or a change in a feedback time of an ultrasonic reflected wave. For example, the proximity sensor 5 detects that the display 2A is brought close to the face. The illuminance sensor 4 and the proximity sensor 5 may be configured as one sensor. The illuminance sensor 4 may be used as a proximity sensor.

  The communication unit 6 communicates wirelessly. The wireless communication standards supported by the communication unit 6 include, for example, cellular phone communication standards such as 2G, 3G, and 4G, and short-range wireless communication standards. Cellular phone communication standards include, for example, LTE (Long Term Evolution), W-CDMA (Wideband Code Division Multiple Access), WiMAX (Worldwide PDA), WiMAX (Worldwide Interoperability Pc) ) (Global System for Mobile Communications), PHS (Personal Handy-phone System), and the like. Examples of short-range wireless communication standards include IEEE 802.11, Bluetooth (registered trademark), IrDA (Infrared Data Association), NFC (Near Field Communication), and WPAN (Wireless Personal Area Network). As a communication standard of WPAN, for example, there is ZigBee (registered trademark). The communication unit 6 may support one or more of the communication standards described above.

  The receiver 7 is a sound output unit. The receiver 7 outputs the sound signal transmitted from the controller 10 as sound. The receiver 7 is used, for example, to output the other party's voice during a call. The microphone 8 is a sound input unit. The microphone 8 converts the user's voice or the like into a sound signal and transmits the sound signal to the controller 10.

  The storage 9 stores programs and data. The storage 9 is also used as a work area for temporarily storing the processing result of the controller 10. The storage 9 may include any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage 9 may include a plurality of types of storage media. The storage 9 may include a combination of a storage medium such as a memory card, an optical disk, or a magneto-optical disk and a storage medium reader. The storage 9 may include a storage device used as a temporary storage area such as a RAM (Random Access Memory).

  The programs stored in the storage 9 include an application executed in the foreground or the background, and a control program (not shown) that supports the operation of the application. An application executed in the foreground displays a screen on the display 2A, for example. The control program includes an OS, for example. The application and the basic program may be installed in the storage 9 via wireless communication by the communication unit 6 or a non-transitory storage medium.

  The storage 9 stores a line-of-sight position detection program 9A, a display position adjustment program 9B, a telephone application 9C, setting data 9Z, and the like.

  The line-of-sight position detection program 9A provides a function for detecting the line-of-sight position of the user with respect to the touch screen display 2. The line-of-sight position corresponds to a position on the touch screen display 2 when a line of sight extending linearly starting from the center position of the pupil in the user's eyeball intersects the touch screen display 2.

  The line-of-sight position detection program 9A detects the line-of-sight position, for example, by applying an algorithm based on the corneal reflection method to the user's image. Specifically, the line-of-sight position detection program 9 </ b> A irradiates infrared rays from the infrared irradiation unit 13 when acquiring a user image. The line-of-sight position detection program 9 </ b> A specifies the position of the pupil and the position of infrared corneal reflection from the acquired user image. The line-of-sight position detection program 9A specifies the direction of the line of sight of the user based on the positional relationship between the position of the pupil and the position of infrared corneal reflection. The line-of-sight position detection program 9A determines that the direction of the line of sight is closer to the corner of the eye if the position of the pupil is closer to the corner of the eye than the position of corneal reflection, and if the position of the pupil is closer to the front of the eye than the position of corneal reflection. The direction of the line of sight is determined to be the eye side. The line-of-sight position detection program 9A calculates the distance between the user's eyeball and the touch screen display 2 based on the size of the iris, for example. The gaze position detection program 9A starts from the pupil from the center position of the pupil in the user's eyeball based on the direction of the user's gaze and the distance between the user's eyeball and the touch screen display 2. A visual line position that is a position on the touch screen display 2 when the visual line intersects the touch screen display 2 is detected.

  In the above example, the case has been described in which the line-of-sight position detection program 9A detects the line-of-sight position of the user by executing processing using the cornea reflection method, but the present invention is not limited to this example. For example, the line-of-sight position detection program 9A may detect the line-of-sight position by executing an image recognition process on the user's image. For example, the line-of-sight position detection program 9A extracts a predetermined region including the user's eyeball from the user's image, specifies the line-of-sight direction based on the positional relationship between the eye and the iris, The line-of-sight position is detected based on the distance from the user's eyeball to the touch screen display 2. Alternatively, the line-of-sight position detection program 9A accumulates a plurality of eyeball images as a reference image when the user is browsing various display areas on the touch screen display 2, and the reference image and the determination target are stored. The line-of-sight position is detected by collating the acquired image of the user's eyeball. The line-of-sight position detection program 9A is an example of a line-of-sight position detection unit.

  The display position adjustment program 9B provides a function for changing the display position of the object displayed on the touch screen display 2 based on the line-of-sight position detected by the line-of-sight position detection program 9A. Specifically, the display position adjustment program 9B executes the following processes. That is, the display position adjustment program 9B determines the object displayed at the first line-of-sight position detected by the line-of-sight position detection program 9A among the plurality of objects displayed on the touch screen display 2 as the target object. When the acceleration sensor 15 detects a horizontal movement with respect to the main surface of the touch screen display 2, the display position adjustment program 9B detects the line of sight detected by the line-of-sight position detection program 9A after detecting the movement in the horizontal direction. The position is determined as the second line-of-sight position. The display position adjustment program 9B changes the display position of the target object based on the second line-of-sight position when the distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than the threshold value.

  The display position adjustment program 9B may further provide a function for changing the display position of the target object based on the second line-of-sight position when the acceleration of movement detected by the acceleration sensor 15 is equal to or greater than the threshold value. Good.

  The telephone application 9C provides a telephone call function for a telephone call by wireless communication. The functions provided by the telephone application 9C include a function for displaying a screen for executing a call on the touch screen display 2.

  The setting data 9Z includes various data used for processing executed based on the functions of the control program, the line-of-sight position detection program 9A, and the display position adjustment program 9B. The setting data 9Z may include, for example, various data related to the user's eyeball. The setting data 9Z may include, for example, threshold information for determining the distance between the first line-of-sight position and the second line-of-sight position, a threshold for determining movement acceleration, and the like.

  The controller 10 is an arithmetic processing device. The arithmetic processing unit includes, for example, a CPU (Central Processing Unit), an SoC (System-on-a-Chip), an MCU (Micro Control Unit), an FPGA (Field-Programmable Gate Array), and a coprocessor. It is not limited. The controller 10 controls various operations of the smartphone 1 to realize various functions. The controller 10 is an example of a control unit.

  Specifically, the controller 10 executes instructions included in the program stored in the storage 9 while referring to the data stored in the storage 9 as necessary. And the controller 10 controls a function part according to data and a command, and implement | achieves various functions by it. The functional unit includes, for example, the display 2A, the communication unit 6, the microphone 8, the speaker 11, and the infrared irradiation unit 13, but is not limited thereto. The controller 10 may change the control according to the detection result of the detection unit. Examples of the detection unit include, but are not limited to, the touch screen 2B, the button 3, the illuminance sensor 4, the proximity sensor 5, the microphone 8, the camera 12, the acceleration sensor 15, and the azimuth sensor 16.

  The controller 10 executes the process of detecting the line-of-sight position by executing the line-of-sight position detection program 9A. The controller 10 executes the following process by executing the display position adjustment program 9B. That is, the controller 10 determines the object displayed at the first line-of-sight position among the plurality of objects displayed on the touch screen display 2 as the target object. When a movement in the horizontal direction with respect to the main surface of the touch screen display 2 is detected by the acceleration sensor 15, the controller 10 determines the line-of-sight position detected after detection of the movement in the horizontal direction as the second line-of-sight position. To do. When the distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than the threshold, the controller 10 changes the display position of the target object based on the second line-of-sight position.

  Hereinafter, the example of the process performed by the smart phone 1 which concerns on Embodiment 1 is demonstrated using FIGS. 2-6 is a figure for demonstrating the example of the process performed by the smart phone 1 which concerns on Embodiment 1. FIG.

  The processing illustrated in FIGS. 2 to 6 is executed in a situation where there is no change in the line-of-sight direction directed from the user to the touch screen display 2 before and after the movement of the smartphone 1 in the horizontal direction. The processing shown in FIGS. 2 to 6 is performed by changing the display position where an object on the extended line in the user's line of sight is displayed on the touch screen display 2 before and after the smartphone 1 is moved in the horizontal direction. The object is to realize a process of adjusting so as not to deviate from the line-of-sight direction.

  As illustrated in FIG. 2, the smartphone 1 detects a line-of-sight position and detects a first line-of-sight position VP1. Then, the smartphone 1 determines the object T1 displayed at the first line-of-sight position VP1 as a target object (step S11).

  Subsequently, when the smartphone 1 detects a movement in the horizontal direction d1 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP2 to be detected after detecting the movement in the horizontal direction as the second line-of-sight position. (Step S12a). Subsequently, when the distance between the first line-of-sight position VP1 and the second line-of-sight position VP2 is equal to or greater than the threshold, the smartphone 1 changes the display position of the object T1 that is the target object to the second line-of-sight position VP2. It changes based on (step S12a).

  FIG. 3 shows changes in the line-of-sight position corresponding to steps S11 to S12a shown in FIG. When the smartphone 1 is moved in the horizontal direction d1 with no change in the line-of-sight direction directed from the user to the touch screen display 2, it is detected by the smartphone 1 even if there is no change in the user's line-of-sight direction. The line-of-sight position changes before and after the movement. That is, as shown in FIG. 3, for example, the first line-of-sight position VP1 moves by the moving distance L. As a result, as shown in FIG. 3, the line-of-sight position detected by the smartphone 1 is the second line-of-sight position VP2. Therefore, the smartphone 1 realizes a process of adjusting the display position of the object so as not to be removed from the user's line-of-sight direction by changing the display position of the object T1 based on the second line-of-sight position VP2.

  As described above, the smartphone 1 detects the line of sight detected on the touch screen display 2 by the movement of the smartphone 1 under the condition that the line of sight directed from the user to the touch screen display 2 is not changed. When the position is changed (when the distance between the first line-of-sight position VP1 and the second line-of-sight position VP2 is equal to or greater than the threshold), the display position of the target object is set to the horizontal movement of the smartphone 1. Adjust so that it does not deviate from the user's line of sight before and after On the contrary, when the movement of the target object due to the movement of the smartphone 1 occurs, for example, when the movement occurs in the user's line-of-sight direction, for example, the user's line-of-sight direction moves in a direction following the target object ( When the distance between the first line-of-sight position VP1 and the second line-of-sight position VP2 is less than the threshold value), the display position of the target object is not adjusted. In this way, the smartphone 1 ensures the purpose of adjusting the display position of the target object so as not to deviate from the user's line-of-sight direction before and after the movement of the smartphone 1 in the horizontal direction.

  For example, in an environment in which vibration is likely to act on a smartphone, such as while riding on a vehicle or a construction site, it is expected that a movement unintended by the user will occur in the smartphone 1 during operation of the smartphone 1. For such a situation, the smartphone 1 cancels the movement of the target object due to the movement of the smartphone 1 by changing the display position of the target object. For this reason, the smart phone 1 can reduce a user's erroneous input with respect to a target object (software key etc.). Moreover, since it is not necessary for the user to follow the line of sight to the target object (software key or the like) every time the smartphone 1 moves, an effect of reducing eye fatigue can be considered.

  Similarly to step S11 to step S12a, when the smartphone 1 detects movement in the horizontal direction d2 with respect to the main surface of the touch screen display 2, the smartphone 1 sets the line-of-sight position VP3 to be detected after detection of movement in the horizontal direction. The second line-of-sight position is determined (steps S11 to S12b). Subsequently, when the distance between the first line-of-sight position VP1 and the second line-of-sight position VP3 is equal to or greater than the threshold, the smartphone 1 changes the display position of the object T1 that is the target object to the second line-of-sight position VP3. It changes based on (step S12b).

  FIG. 4 shows an example of processing when the smartphone 1 moves in a direction different from the example shown in FIG. As illustrated in FIG. 4, the smartphone 1 detects a line-of-sight position and detects a first line-of-sight position VP4. Then, the smartphone 1 determines the object T1 displayed at the first line-of-sight position VP4 as a target object (Step S21).

  Subsequently, when the smartphone 1 detects a movement in the horizontal direction d3 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP5 detected after detecting the movement in the horizontal direction as the second line-of-sight position. (Step S22a). Subsequently, when the distance between the first line-of-sight position VP4 and the second line-of-sight position VP5 is greater than or equal to the threshold, the smartphone 1 changes the display position of the object T1 that is the target object to the second line-of-sight position VP5. It changes based on (step S22a).

  Similarly to step S21 to step S22a described above, when the smartphone 1 detects movement in the horizontal direction d4 with respect to the main surface of the touch screen display 2, the smartphone 1 sets the line-of-sight position VP6 to be detected after detection of movement in the horizontal direction. The second line-of-sight position is determined (steps S21 to S22b). Subsequently, when the distance between the first line-of-sight position VP4 and the second line-of-sight position VP6 is equal to or greater than the threshold, the smartphone 1 changes the display position of the object T1 that is the target object to the second line-of-sight position VP6. It changes based on (step S22b).

  5 and 6 show an example of processing that makes an object more specific. 5 and 6 show an example in which a telephone application screen 50 is displayed on the touch screen display 2. On the screen 50, a plurality of keys are displayed as objects for the user to input. A function for realizing a process of generating a duplicate of an object as a target object from a plurality of objects included in the screen 50 and displaying the duplicate on the screen 50 in order to realize the processing shown in FIGS. May be mounted on the smartphone 1 in advance. The present invention is not limited to the examples shown in FIGS. 5 and 6, and a process of actually separating a target object (for example, one of the keys) from a plurality of objects included in the screen 50 and displaying it on the screen 50 is realized. The function for this may be mounted in the smart phone 1 beforehand.

  As illustrated in FIG. 5, the smartphone 1 detects a line-of-sight position and detects a first line-of-sight position VP7. Then, the smartphone 1 determines the object T2 displayed at the first line-of-sight position VP7 as a target object (step S31).

  Subsequently, when the smartphone 1 detects the movement in the horizontal direction d1 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP8 detected after the movement in the horizontal direction is detected as the second line-of-sight position. (Step S32). Subsequently, the smartphone 1 changes the display position of the object T2 that is the target object based on the second line-of-sight position VP8 (step S32).

  On the other hand, as illustrated in FIG. 6, the smartphone 1 detects the line-of-sight position before detecting movement in the horizontal direction, and as a result, detects the first line-of-sight position VP7. Then, the smartphone 1 determines the object T2 displayed at the first line-of-sight position VP7 as a target object (step S41).

  Subsequently, when the smartphone 1 detects movement in the horizontal direction d2 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP9 detected after detection of movement in the horizontal direction as the second line-of-sight position. (Step S42). Subsequently, the smartphone 1 changes the display position of the object T2 that is the target object based on the second line-of-sight position VP9 (step S42).

  For example, as illustrated in FIG. 5, when the distance between the first line-of-sight position VP <b> 7 and the second line-of-sight position VP <b> 8 is equal to or greater than the threshold value, the smartphone 1 has a number “3” key displayed on the screen 50. The display position of (object) is adjusted so as not to deviate from the user's line of sight before and after the smartphone 1 is moved in the horizontal direction. In the example shown in FIGS. 5 and 6, a case where a copy of an object to be a target object is displayed on the screen 50 is not limited to this example. For example, from among a plurality of objects included in the screen 50, A part of the object that is the target object may be actually separated and displayed on the screen 50.

  The flow of processing by the smartphone 1 according to the first embodiment will be described with reference to FIG. FIG. 7 is a flowchart illustrating a process flow by the smartphone 1 according to the first embodiment. The processing shown in FIG. 7 is realized by the controller 10 executing a line-of-sight position detection program 9A, a display position adjustment program 9B, and the like stored in the storage 9.

  The speaker 11 is a sound output unit. The speaker 11 outputs the sound signal transmitted from the controller 10 as sound. The speaker 11 is used for outputting a ring tone and music, for example. One of the receiver 7 and the speaker 11 may also function as the other.

  The camera 12 converts the captured image into an electrical signal. The camera 12 is an in camera that captures an object facing the display 2A. For example, the camera 12 captures an image of the eyeball of the user who turns his / her line of sight toward the touch screen display 2. The camera 12 may be mounted on the smartphone 1 as a camera unit that can be used by switching between the in-camera and the out-camera. The camera 12 is an example of an image acquisition unit. The infrared irradiation unit 13 irradiates infrared rays in synchronization with photographing performed by the camera 12.

  The connector 14 is a terminal to which another device is connected. The connector 14 may be a general-purpose terminal such as USB (Universal Serial Bus), HDMI (registered trademark) (High-Definition Multimedia Interface), Light Peak (Thunderbolt (registered trademark)), and an earphone microphone connector. . The connector 14 may be a dedicated terminal such as a dock connector. Devices connected to the connector 14 include, but are not limited to, external storage, speakers, and communication devices, for example.

  The acceleration sensor 15 detects the direction and magnitude of acceleration acting on the smartphone 1. For example, the acceleration sensor 15 detects a movement in the horizontal direction with respect to the main surface (image display surface) of the touch screen display 2 based on the direction of acceleration. The acceleration sensor 15 is an example of a movement detection unit. The direction sensor 16 detects, for example, the direction of geomagnetism, and detects the direction (azimuth) of the smartphone 1 based on the direction of geomagnetism.

  The smartphone 1 may include a GPS receiver and a vibrator in addition to the above functional units. The GPS receiver receives a radio signal in a predetermined frequency band from a GPS satellite, demodulates the received radio signal, and sends the processed signal to the controller 10. The vibrator vibrates a part or the whole of the smartphone 1. The vibrator includes, for example, a piezoelectric element or an eccentric motor in order to generate vibration. Although not shown in FIG. 1, functional units such as a battery that are naturally used to maintain the functions of the smartphone 1 are mounted on the smartphone 1.

  The flow of processing by the smartphone 1 according to the first embodiment will be described with reference to FIGS. 7 and 8. 7 and 8 are flowcharts showing a flow of processing by the smartphone 1 according to the first embodiment. The processing illustrated in FIGS. 7 and 8 is realized by the controller 10 executing the line-of-sight position detection program 9A, the display position adjustment program 9B, and the like stored in the storage 9.

  As illustrated in FIG. 7, the controller 10 determines the target object based on the detected first line-of-sight position (step S101).

  Subsequently, the controller 10 determines whether movement in the horizontal direction is detected with respect to the main surface of the display unit (touch screen display 2) (step S102).

  As a result of the determination, when the movement in the horizontal direction with respect to the main surface of the display unit is not detected (No at Step S102), the controller 10 repeats the determination at Step S102.

  On the other hand, when the movement in the horizontal direction is detected with respect to the main surface of the display unit as a result of the determination (step S102, Yes), the controller 10 determines the line-of-sight position detected after the detection of the movement in the horizontal direction. The second line-of-sight position is determined (step S103).

  Subsequently, the controller 10 determines whether or not the distance between the first line-of-sight position and the second line-of-sight position is greater than or equal to a threshold (step S104).

  As a result of the determination, when the distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than the threshold value (step S104, Yes), the controller 10 determines the display position of the target object as the second position. Is changed based on the line-of-sight position (step S105), and the process shown in FIG.

  On the other hand, as a result of the determination, when the distance between the first line-of-sight position and the second line-of-sight position is not equal to or greater than the threshold (No in step S104), the controller 10 ends the process illustrated in FIG. To do.

  In the process illustrated in FIG. 7, the smartphone 1 may change the display position of the target object when the detected acceleration of movement is greater than or equal to the threshold value. Hereinafter, the flow of processing by the smartphone 1 in this case will be described with reference to FIG. The processing shown in FIG. 8 is different from the processing shown in FIG.

  As shown in FIG. 8, the controller 10 determines the target object based on the detected first line-of-sight position (step S201).

  Subsequently, the controller 10 determines whether movement in the horizontal direction is detected with respect to the main surface of the display unit (touch screen display 2) (step S202).

  As a result of the determination, when the horizontal movement with respect to the main surface of the display unit is not detected (No at Step S202), the controller 10 repeats the determination at Step S202.

  On the other hand, as a result of the determination, when the movement in the horizontal direction is detected with respect to the main surface of the display unit (step S202, Yes), the acceleration of the movement in the horizontal direction is equal to or greater than the threshold value. Is determined (step S203).

  When the acceleration of the movement in the horizontal direction is equal to or greater than the threshold value as a result of the determination (step S203, Yes), the controller 10 sets the line-of-sight position detected after the movement in the horizontal direction as the second line-of-sight position. Confirm (step S204).

  Subsequently, the controller 10 determines whether the distance between the first line-of-sight position and the second line-of-sight position is greater than or equal to a threshold value (step S205).

  As a result of the determination, when the distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than the threshold (Yes in step S205), the controller 10 determines the display position of the target object as the second position. Is changed based on the line-of-sight position (step S206), and the process shown in FIG.

  On the other hand, if the determination result shows that the distance between the first line-of-sight position and the second line-of-sight position is not greater than or equal to the threshold (No at step S205), the controller 10 ends the process shown in FIG. To do.

  In step S203, if the result of determination is that the acceleration of movement in the horizontal direction is not greater than or equal to the threshold value (step S203, No), the controller 10 ends the processing shown in FIG.

  In the process shown in FIG. 8, it is possible to determine whether the movement is intended by the user by determining whether the acceleration of movement is equal to or greater than a threshold value.

(Embodiment 2)
In the first embodiment, the display position of an object other than the target object displayed on the touch screen display 2 may be adjusted in accordance with the change in the display position of the target object. Hereinafter, an example of processing by the smartphone 1 in this case will be described with reference to FIGS. 9 and 10. 9 and 10 are diagrams for explaining an example of processing executed by the smartphone 1 according to the second embodiment. In the example illustrated in FIGS. 9 and 10, the target object corresponds to the key “3” among the plurality of keys (objects) included in the screen 50, and the remaining keys other than the number “3” are the target objects. Applies to objects other than.

  FIG. 9 shows an example of processing when the display position of an object other than the target object is changed simultaneously with the change of the display position of the target object.

  As illustrated in FIG. 9, the smartphone 1 detects a line-of-sight position and detects a first line-of-sight position VP7. Then, the smartphone 1 determines the object T2 displayed at the first line-of-sight position VP7 as a target object (step S51).

  Subsequently, when the smartphone 1 detects the movement in the horizontal direction d1 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP8 detected after the movement in the horizontal direction is detected as the second line-of-sight position. (Step S52). Subsequently, when the distance between the first line-of-sight position VP7 and the second line-of-sight position VP8 is equal to or greater than the threshold, the smartphone 1 changes the display position of the object T2 that is the target object to the second line-of-sight position VP8. Based on the change (step S52). When changing the display position of the target object T2, the smartphone 1 maintains the positional relationship between the display position of the object T3 (remaining keys) other than the target object T2 and the target object T2 before the display position change. (Step S52).

  Since the smartphone 1 changes the display position of an object other than the target object, for example, a series of operations via a plurality of objects can be secured without delay, such as a key operation.

  FIG. 10 shows an example of processing when the display position of an object other than the target object is changed, triggered by detection of a predetermined operation on the target object after the display position is changed. In order to realize the processing shown in FIG. 10, it is possible to accept processing for displaying a copy of the target object as separated from the screen 50 from among a plurality of objects included in the screen 50 and operations for copying the target object. A function for realizing the processing to be in a proper state may be mounted in the smartphone 1 in advance.

  As illustrated in FIG. 10, the smartphone 1 detects a line-of-sight position and detects a first line-of-sight position VP7. Then, the smartphone 1 determines the object T2 displayed at the first line-of-sight position VP7 as a target object (Step S61).

  Subsequently, when the smartphone 1 detects the movement in the horizontal direction d1 with respect to the main surface of the touch screen display 2, the smartphone 1 determines the line-of-sight position VP8 detected after the movement in the horizontal direction is detected as the second line-of-sight position. (Step S62). Subsequently, when the distance between the first line-of-sight position VP7 and the second line-of-sight position VP8 is equal to or greater than the threshold, the smartphone 1 changes the display position of the object T2 that is the target object to the second line-of-sight position VP8. Based on the change (step S62).

  Subsequently, when the smartphone 1 detects a predetermined operation on the target object T2 after the change of the display position, the display position of the object T3 (remaining keys) other than the target object T2 is triggered by the operation, It changes so that the positional relationship with target object T2 before a display position change may be maintained (step S63-step S64).

  As shown in FIG. 10, the smartphone 1 changes the display position of an object other than the target object, and thus prevents erroneous input to the target object, and for example, a plurality of objects after an operation on the target object, such as a key operation. It can be ensured that a series of operations through the network is performed without delay.

  In the example illustrated in FIG. 10, the smartphone 1 may perform control so that an operation on an object other than the target object is invalid until the change of the display position of the object T2 that is the target object is completed. In the example shown in FIG. 10, a case where a copy of an object to be a target object is displayed on the screen 50 is shown. However, the present invention is not limited to this example. For example, a target object is selected from a plurality of objects included in the screen 50. A part of the objects may be actually separated and displayed on the screen 50.

(Other embodiments)
In the first embodiment and the second embodiment described above, the processing based on the assumption that the user's line-of-sight direction does not move (that is, the movement of the eye is fixed) before and after the movement of the smartphone 1 has been described. The mode for executing this processing is often used in an environment of vibration with a relatively short cycle and small amplitude (that is, vibration that the human eye cannot follow). On the other hand, in a vibration environment with a relatively large amplitude, there is a high possibility that the target object will be out of the range of the touch screen display 2. In addition, in a vibration environment with a relatively long cycle, the user's line-of-sight direction can be moved following the movement of the smartphone 1 (target object). In this way, in an environment where the amplitude is relatively large or an environment where the vibration is relatively long, a mode for executing processing for causing the position of the target object to follow the second line-of-sight position may be set.

  FIG. 11 is a diagram for explaining the processing of the other embodiment described above. For example, as shown in FIG. 11, the distance between the first line-of-sight position VP1 and the second line-of-sight position VP10 is greater than or equal to the threshold before and after the smartphone 1 moves in the horizontal direction d1 with respect to the main surface of the touch screen display 2. If there is an opening, the smartphone 1 changes the display position of the target object T1 by following the second line-of-sight position VP10 (steps S71 to S72a). Here, the second line-of-sight position VP10 corresponds to the line-of-sight direction of the user who has moved in the left direction of the touch screen display 2 (the same direction as d1). Similarly, before and after the smartphone 1 moves in the horizontal direction d2 with respect to the main surface of the touch screen display 2, when the distance between the first line-of-sight position VP1 and the second line-of-sight position VP11 is larger than the threshold, The smartphone 1 changes the display position of the target object T1 by following the second line-of-sight position VP11 (steps S71 to S72b). Here, the second line-of-sight position VP10 corresponds to the line-of-sight direction of the user who has moved downward in the touch screen display 2 (a downward direction perpendicular to d2). The process flow shown in FIG. 11 is the same as the process flow shown in FIGS. 7 and 8 except that the display position of the target object is changed following the second line-of-sight position. As described above, the smartphone 1 according to the present application performs processing when the user's line-of-sight direction does not move (FIGS. 7 and 8), and processing when the user's line-of-sight direction moves (FIG. 11). It can be selected and used according to the state of vibration acting on the smartphone 1.

  In said embodiment, although the smart phone 1 was demonstrated as an example of the apparatus which concerns on an attached claim, the apparatus which concerns on an attached claim is not limited to the smart phone 1. FIG. The device according to the appended claims may be a device other than a smartphone as long as it is an electronic device having a touch screen.

  The characterizing embodiments have been described in order to fully and clearly disclose the technology according to the appended claims. However, the appended claims should not be limited to the above-described embodiments, but all modifications and alternatives that can be created by those skilled in the art within the scope of the basic matters shown in this specification. Should be embodied by a possible configuration.

DESCRIPTION OF SYMBOLS 1 Smart phone 2 Touch screen display 2A Display 2B Touch screen 3 Button 4 Illuminance sensor 5 Proximity sensor 6 Communication unit 7 Receiver 8 Microphone 9 Storage 9A Gaze position detection program 9B Display position adjustment program 9C Telephone application 9Z Setting data 10 Controller 11 Speaker 12 Camera 13 Infrared irradiation part 14 Connector 15 Acceleration sensor 16 Direction sensor

Claims (5)

A display unit that displays a plurality of objects for input operation on the main surface;
A movement detector for detecting movement;
An image acquisition unit for acquiring images;
A line-of-sight position detection unit that detects a line-of-sight position of a user corresponding to any of the plurality of objects based on the image acquired by the image acquisition unit;
Of the plurality of objects, the object displayed at the first line-of-sight position detected by the line-of-sight position detection unit is determined as a target object,
When a movement in the horizontal direction with respect to the main surface of the display unit is detected by the movement detection unit, the line-of-sight position detected by the line-of-sight position detection unit after the movement in the horizontal direction is detected is a second line-of-sight position. As
A control unit that changes a display position of the target object based on the second line-of-sight position when a distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than a threshold value. Mobile device.   The said control part changes the display position of the said target object based on a said 2nd visual line position, when the acceleration of the movement detected by the said movement detection part is more than a predetermined threshold value. Mobile device.   The said control part changes the display position of objects other than the said target object so that the positional relationship with the said target object before a display position change may be maintained, when changing the display position of the said target object. Or the portable terminal of 2.   The control unit maintains a positional relationship between a display position of an object other than the target object and the target object before the display position is changed, triggered by detection of a predetermined operation on the target object after the display position is changed. The mobile terminal according to claim 1 or 2, which is modified as follows. A display unit that displays a plurality of objects for input operation on the main surface;
A movement detector for detecting movement;
An image acquisition unit for acquiring images;
A control method to be executed by a mobile terminal having a gaze position detection unit that detects a gaze position of a user corresponding to any of the plurality of objects based on an image acquired by the image acquisition unit,
Determining an object displayed at a first line-of-sight position detected by the line-of-sight position detection unit among the plurality of objects as a target object;
When movement in the horizontal direction with respect to the main surface of the display unit is detected by the movement detection unit, the line-of-sight position detected by the line-of-sight position detection unit after the detection of vibration in the horizontal direction is the second line-of-sight position. Step to confirm as
A step of changing a display position of the target object based on the second line-of-sight position when a distance between the first line-of-sight position and the second line-of-sight position is equal to or greater than a threshold value. Method.

Images