TWI796883B - Method, electronic device, and user interface for user interface control - Google Patents

Abstract

This application discloses a user interface control method, an electronic device and a user interface. The user interface comprises at least one first functional area, and the method comprises the following steps. Acquiring eyeball movement information of a user. Selecting a corresponding first functional area according to the eyeball motion information. The first functional area comprises at least one second functional area. Collecting sensory input information of the user; and selecting the corresponding second functional area according to the sensory input information of the user. Executing the executable function corresponding to the second functional area.

Description

User interface control method, electronic device and user interface

The present application relates to the technical field of user interface, and specifically relates to a user interface control method, an electronic device and a user interface.

Along with the development of economy and the advancement of science and technology, automobiles are used more and more widely in people's lives as a common means of transportation. Whether the driver is in a state of concentration while driving the vehicle has a very serious impact on safe driving. Therefore, driver distraction should be avoided as much as possible. The driver may encounter incoming calls or voice messages during driving, and may need to answer or reply. At present, the process for the driver to answer or reply to messages during driving is relatively complicated, which will cause the driver to be distracted and unable to understand the road conditions in time, which may lead to accidents.

In view of this, the application provides a user interface control method, an electronic device and a user interface. The ability to select the corresponding functional area according to the eye movement information, and then select the functions that can be executed in the functional area can improve the manipulability of the user interface.

According to the user interface control method of the present application, the user interface includes at least one first functional area, and the method includes: acquiring user's eye movement information; selecting the corresponding first functional area according to the eye movement information, so The first functional area includes at least one second functional area; The user's sensory input information; select the corresponding second functional area according to the user's sensory input information, and execute the executable function corresponding to the second functional area.

The electronic device of the present application includes an eye tracker, a gesture recognition module, a speech recognition module, a display screen, and a processor; the processor is used to execute the user interface control method as described above; the eye tracker is used to obtain Eye movement information; the gesture recognition module is used to recognize gesture information; the voice recognition module is used to recognize voice information.

The user interface of the present application is applied to a display screen, and the display screen is connected to an eye tracking module through communication, and the user interface includes the user interface displayed when the above-mentioned electronic device executes the user interface control method .

100: Electronic device

110: display screen

111: User Interface

120: Eye tracker

130: Gesture recognition module

140:Speech recognition module

150: Processor

200: car

210: Vehicle electronic control device

220: Controller LAN bus

230: Vehicle system

1111-1114: the first functional area

11121-11126: Second functional area

S100-S331: Steps

FIG. 1 is a schematic diagram of an electronic device module provided by an embodiment of the present application.

Fig. 2 is a schematic diagram of a car module provided by an embodiment of the present application.

FIG. 3 is a schematic diagram of a user interface provided by an embodiment of the present application.

FIG. 4 is a schematic diagram of a user interface provided by another embodiment of the present application.

FIG. 5 is a schematic diagram of gesture recognition provided by an embodiment of the present application.

FIG. 6 is a schematic flowchart of a user interface control method provided by an embodiment of the present application.

FIG. 7 is a schematic flowchart of a user interface control method provided by another embodiment of the present application.

FIG. 8 is a schematic flowchart of a user interface control method provided by another embodiment of the present application.

In order to more clearly understand the above purpose, features and advantages of the present application, the present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. Need to explain, in the case of no conflict, the The embodiments of the application and the features in the embodiments can be combined with each other. A lot of specific details are set forth in the following description to facilitate a full understanding of the application, and the described embodiments are only some of the embodiments of the application, not all of them.

It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than in the flowchart. The methods disclosed in the embodiments of the present application include one or a plurality of steps or actions for realizing the methods. The method steps and/or actions may be interchanged with one another without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.

Some embodiments of the application will be described in detail below in conjunction with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

Among the various on-board equipment of the car, the central control screen occupies an important position, and the car owner can perform multiple operations on the car through the central control screen. The existing central control screen is mainly divided into two categories: button operation and screen operation, but no matter what kind of operation method, the central control screen is developing with the trend of large screen, integrating and displaying more car status and multimedia information. Therefore, the central control The importance of screens in driving is also increasing. As the size of the central control screen becomes larger, manipulating the central control screen while the car is driving will bring safety hazards.

At present, most vehicle-mounted systems integrate eye-tracking technology to control the central control screen. However, as the functions of the central control screen become more and more complex, it is difficult to use eye tracking technology to perform precise control.

The user interface control method, user interface and device provided by the embodiment of the present application can select the corresponding functional area according to the eye movement information, and then select the executable functions in the functional area by voice recognition or gesture recognition, which can improve the user experience. The manipulability of the interface.

Some embodiments of the application will be described in detail below in conjunction with the accompanying drawings. In the case of no conflict, the following embodiments and features in the embodiments can be combined with each other.

FIG. 1 is an electronic device 100 provided by an embodiment of the present application. As shown in FIG. 1 , the electronic device 100 includes a display screen 110 , an eye tracker 120 , a gesture recognition module 130 , a speech recognition module 140 and a processor 150 . Wherein, the display screen 110 is used for displaying a user interface 111 .

It can be understood that the electronic device 100 realizes the display function through the GPU, the display screen 110 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 110 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. The electronic device 100 may include one or more GPUs that execute program instructions to generate or change display information.

It can be understood that the display screen 110 is used to display images, videos and the like. The display screen 110 includes a display panel. The display panel can be a liquid crystal display (liquid crystal display, LCD), an organic light-emitting diode (organic light-emitting diode, OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix) organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device may include 1 or N display screens 110 , where N is a positive integer greater than 1.

It can be understood that the display screen 110 is used for displaying a user interface (user interface, UI) 111 . When it can be understood, the user interface 111 is a medium interface for interaction and information exchange between the application program or the operating system and the user, and it realizes the conversion between the internal form of the information and the form acceptable to the user. The user interface 111 of the application program is an original code written in a specific computer language such as java and extensible markup language (XML). The original code of the interface is parsed and rendered on the terminal device, and finally presented as a Content that can be recognized by the user, such as pictures, text, buttons and other control items. Controls, also known as widgets, are the basic elements of the user interface 111. Typical control items include toolbars, menubars, textboxes, and buttons. , scrollbar (scrollbar), picture and text. The properties and content of the control items in the interface are defined by tags or nodes, such as XML by Nodes such as <Textview>, <ImgView>, <VideoView> define the control items included in the interface. A node corresponds to a control item or attribute in the interface. After the node is parsed and rendered, it is presented as the content visible to the user. In addition, the interfaces of many applications, such as hybrid applications, usually include webpages. A web page, also known as a page, can be understood as a special control item embedded in an application programming interface. A web page is a source code written in a specific computer language, such as hyper text markup language (HTML) , cascading style sheets (cascading style sheets, CSS), java script (JavaScript, JS), etc., the source code of the web page can be loaded and displayed as recognizable by the user by a browser or a web page display element similar to the browser function the content. The specific content contained in the webpage is also defined by the tags or nodes in the original code of the webpage. For example, HTML uses <p>, <img>, <video>, <canvas> to define the elements and attributes of the webpage.

It can be understood that the commonly used expression form of the user interface 111 is a graphical user interface (graphic user interface, GUI), which refers to the user interface 111 related to computer operation displayed in a graphical manner. It can be an interface element such as an icon, window, control item displayed on the display screen 110 of the electronic device 100, wherein the control item can include an icon, button, menu, tab, text box, dialog box, status bar , Tour bar, Widget and other visual interface elements.

It can be understood that the eye tracker 120 may include a camera and an infrared device (such as an infrared emitter) to detect the user's eye movements, such as eye gaze direction, blinking operation, gaze operation, etc., thereby realizing eye tracking (eye tracking). Applications are not limited here.

It can be understood that after the gesture recognition module 130 detects the user's gestures through an optical sensor (such as a camera, millimeter-wave radar, lidar, etc.), the user's gesture model stored in the electronic device 100 can be compared with The user's gesture information is compared to determine whether the two match, and when it is determined that they match, the corresponding control is executed.

It can be understood that the speech recognition module 140 can use a microphone and a neural-network (NN) computing processor, by referring to the biological neural network structure, for example, referring to the transmission mode between neurons in the human brain, to realize speech identify.

It can be understood that the processor 150 is used to identify sensory input information to generate control signals. Exemplarily, the sensory input information includes any one or a combination of the following: gesture information, voice information. Wherein, the gesture information includes: motion track information, spatial position information, gesture shape information or pointing information and combinations thereof.

For example, the processor 150 is configured to control the display screen 110 according to the eye movement information, gesture information or voice information collected by the eye tracker 120, the gesture recognition module 130 or the speech recognition module 140 to adjust and display the information in the user interface 111. Display content, and control the car machine in the car to perform corresponding functions.

It can be understood that the electronic device 100 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, and a cell phone, a personal Digital assistant (personal digital assistant, PDA), augmented reality (augmented reality, AR) equipment, virtual reality (virtual reality, VR) equipment, artificial intelligence (artificial intelligence, AI) equipment, wearable equipment, vehicle equipment, smart Household equipment and/or smart city equipment, some embodiments of the present application do not specifically limit the specific type of the electronic device 100 .

FIG. 2 is a car 200 provided by an embodiment of the present application. As shown in FIG. 2 , the car 200 includes a vehicle electronic control device 210 , a controller area network bus (Controller Area Network Bus, CAN Bus) 220 and a vehicle system 230 .

Exemplarily, the in-vehicle system 230 is provided with a central control unit, an in-vehicle camera, a memory, a display screen, a speaker, a microphone, a voice assistant module, and a communication module. The vehicle system 230 is connected to the electronic device 100 through the communication module.

For example, the controller area network bus 220 connects the vehicle system 230 and the vehicle electronic control device 210 to realize the transmission of commands and messages.

Exemplarily, the vehicle electronic control device 210 is integrated with control systems for power, steering, gear, suspension, brake and door.

FIG. 3 is a user interface 111 provided by an embodiment of the present application. As shown in FIG. 3 , the user interface 111 includes a plurality of first functional areas. For example, four first functional areas 1111 , 1112 , 1113 and 1114 are shown in FIG. 3 .

It can be understood that when the user interface 111 is in the open state, the eye tracker 120 continuously detects the gaze range of the human eye. When the eye tracker 120 detects that the human eye is looking at the user interface 111 , the processor 150 further determines whether the line of sight falls on the first functional area 1111 , 1112 , 1113 , or 1114 .

In some embodiments, the processor 150 numbers the second functional areas within the first functional area, and each of the second functional areas corresponds to at least one executable function.

Please also refer to FIG. 4 . As shown in FIG. 4 , the first functional area 1112 includes second functional areas 11121 , 11122 , 11123 , 11124 , 11125 and 11126 .

For example, when the eye tracker 120 detects that the gaze range of the human eye falls on the first functional area 1112, the processor 150 highlights the first functional area 1112, and numbers the functions available for execution in the first functional area 1112 And display its number, so that the driver can use gestures or voice to control the vehicle to perform related functions. For example, the executable functions in the first functional area 1112 are numbered the second functional area 11121 , 11122 , 11123 , 11124 , 11125 and 11126 . For example, as shown in FIG. 4 , the second functional areas 11121 to 11126 are respectively numbered with digits 1 to 6, and the user can select the second functional areas 11121 to 11126 through digits 1 to 6 . Subsequently, the user's sensory input information is collected, the corresponding second functional area is selected according to the user's sensory input information, and the executable function corresponding to the second functional area is executed. For example, the sensory input information may include gesture information and voice information. The processor 150 can recognize by gesture The module 130 or the speech recognition module 140 obtains the digits 1 to 6 expressed by the user, and then selects the second functional area corresponding to the digits. The processor 150 controls the in-vehicle system 230 to perform corresponding operations according to the functions available in the corresponding second functional area, or transmits control commands to the vehicle electronic control device 210 through the controller area network bus 220 to perform corresponding operations.

For example, when the eye tracker 120 detects that the gaze range of the human eye falls on the first functional area 1112, the processor 150 may also prompt a selection of the first functional area 1112 in the user interface 111, and select the prompt Including any one or combination of the following: highlight prompt, flashing prompt, sound prompt or vibration prompt.

It can be understood that when the processor 150 recognizes the executable function according to the gesture information collected by the gesture recognition module 130, it can recognize the gesture information according to the gesture action, wherein the gesture information includes motion track information, spatial position information or pointing information, Select, move and release the functional area, and then select the number corresponding to the gesture shape according to the gesture shape information.

It can be understood that the gesture recognition module 130 can set specific gesture shapes for some second functional areas. Gesture shapes, such as "V", are used to control the car 200 to perform specific functions through the processor 150 .

The voice message includes any one or combination of the following: the number of the second functional area, the name of the second functional area, the abbreviation of the second functional area, the name of the executable function, and the abbreviation of the executable function , Confirm or Cancel.

Please also refer to FIG. 5 , which is a schematic diagram of gesture recognition provided by an embodiment of the present application.

For example, when the eye tracker 120 detects that the sight range of the human eye falls on the second functional area 1112, the gesture recognition module 130 can recognize the digital sign corresponding to the shape of the user's gesture, and according to the recognized digit The token executes a second functional area with the same digit token. For example, when the user chooses to use the second functional area 11122, the gesture corresponding to "2" can be drawn, so that the processor can 150 controls the car 200 to execute the function corresponding to the second function area 11122 in the first function area 1112 , for example, the seat heating function corresponding to the second function area 11122 shown in FIG. 4 .

It can be understood that since gestures representing digits in different regions may be different, the gesture recognition module 130 may store gestures representing digits in different regions.

For example, when the eye tracker 120 detects that the gaze range of the human eye falls on the second functional area 1112, the processor 150 can recognize the user's voice through the voice recognition module 140, and recognize the corresponding number according to the voice . For example, when the user chooses to use the second functional area 11121 , he can say "one", "the first", "the first function", "the first executable function" and other voices to execute the second functional area 11121 . The voice recognition module 140 can extract the keyword "one" through semantic recognition to select the second functional area 11121 .

It can be understood that the voice recognition module 140 can set specific voices for some second functional areas. For example, when the eye tracker 120 detects that the gaze range of the human eye falls on the user interface 111, the voice recognition module 140 can detect a specific voice, such as "play music", "cut song", etc., to perform music playback , switch songs and other functions.

It can be understood that the voice recognition module 140 acquires a voice control signal according to the voice message, and the voice control signal includes any one of the following or a combination thereof: selecting the second functional area, executing an executable function, confirming the execution of an executable function, canceling the execution of an executable function, or Returning to the previous stage, the processor 150 performs corresponding operations according to the voice control signal.

FIG. 6 is a schematic flowchart of a user interface control method provided by an embodiment of the present application. As shown in Figure 6, the user interface control method at least includes the following steps:

S100: Obtain eye movement information of the user.

It can be understood that the eye tracker 120 is used to acquire the eye movement information of the user. Please refer to FIG. 1 and FIG. 3 for the specific acquisition method, and details are not repeated here.

S200: Select the corresponding first functional area according to the eye movement information.

It can be understood that the processor 150 determines whether the user's line of sight falls on the corresponding first functional area according to the user's eye movement information obtained by the eye tracker 120, and selects the corresponding first functional area. For the specific selection method, please refer to FIG. 1 and FIG. 3 are not repeated here.

S300: Select the corresponding second function area according to the user instruction, and execute the executable function corresponding to the second function area.

It can be understood that, the gesture recognition module 130 or the speech recognition module 140 recognizes the functions that can be executed, so that the processor 150 controls the car 200 to execute the corresponding functions. It can be understood that, for specific selection and execution methods, please refer to FIG. 1 to FIG. 3 together, and details are not repeated here.

FIG. 7 is a schematic flowchart of a user interface control method provided by another embodiment of the present application. As shown in Figure 7, the user interface control method at least includes the following steps:

S310: Apply gesture recognition technology to recognize gesture actions.

It can be understood that the gesture recognition module 130 uses gesture recognition technology to recognize gesture actions. Please refer to FIG. 1 and FIG. 3 for specific recognition methods, and details will not be repeated here.

S320: Identify the gesture control signal according to the gesture action.

It can be understood that the gesture recognition module 130 recognizes the gesture control signal according to the gesture action. For the specific recognition method, please refer to FIG. 1 and FIG. 3 together, which will not be repeated here.

S330: Select a number according to the gesture control signal.

It can be understood that the processor 150 selects the serial number according to the gesture control signal recognized by the gesture recognition module 130 . Please refer to FIG. 1 and FIG. 3 together for the specific selection method, which will not be repeated here.

FIG. 8 is a schematic flowchart of a user interface control method provided by another embodiment of the present application. As shown in Figure 8, the user interface control method at least includes the following steps:

S311: Recognize the voice message by using the voice recognition engine.

It can be understood that the speech recognition module 140 uses a speech recognition engine to recognize speech messages. Please refer to FIG. 1 and FIG. 3 for specific recognition methods, which will not be repeated here.

S321: Acquire a voice control signal according to the voice message.

It can be understood that the speech recognition module 140 obtains the speech control signal according to the recognized speech information. Please refer to FIG. 1 and FIG. 3 for the specific recognition method, and details are not repeated here.

S331: Execute a corresponding operation according to the voice control signal.

It can be understood that the processor 150 executes corresponding operations according to the voice control signal recognized by the voice recognition module 140. Please refer to FIG. 1 and FIG.

It can be understood that the in-vehicle communication method provided by the embodiment of the present application can also be applied to the in-vehicle communication system 10a shown in FIG. This will not be repeated here.

The embodiments of the present application have been described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Within the scope of knowledge of those of ordinary skill in the art, various modifications can be made without departing from the purpose of the present application. Variety. In addition, the embodiments of the present application and the features in the embodiments can be combined with each other under the condition of no conflict.

100: Electronic device

110: display screen

111: User Interface

120: Eye tracker

130: Gesture recognition module

140:Speech recognition module

150: Processor

Claims (10)

A user interface control method, the user interface includes at least one first functional area, the improvement is that the method includes: acquiring the user's eye movement information; selecting the corresponding first functional area according to the eye movement information, so The first functional area includes at least one second functional area; collect sensory input information of the user, the sensory input information includes gesture information, and the gesture information includes gesture shape information; select the corresponding the second functional area, and execute the executable function corresponding to the second functional area. The user interface control method as described in Claim 1, wherein the corresponding second functional area is selected according to the sensory input information of the user, and the executable function corresponding to the second functional area is executed: Identifying the sensory input information to generate a control signal; selecting the second functional area of the corresponding first functional area according to the control signal; wherein the sensory input information also includes voice information. The user interface control method according to Claim 2, wherein the gesture information further includes any one of the following or a combination thereof: motion track information, spatial position information or pointing information. The user interface control method according to claim 2, wherein the voice message includes any one or a combination of the following: the number message of the second functional area, the name message of the second functional area, the first The abbreviation message of the second functional area, the name information of the executable function in the second functional area, the abbreviation message of the executable function in the second functional area, confirmation message or cancellation message. The user interface control method according to claim 2, wherein the control signal includes any one or a combination of the following: selecting the second functional area, executing executable functions in the second functional area, confirming the execution of the A control signal that can execute functions in the second function area, cancel the execution of the executable functions in the second function area, or return to the upper level. The user interface control method according to any one of claims 1 to 5, wherein the method further includes: numbering each of the second function areas, and each of the second function areas corresponds to at least one executable Function. The user interface control method according to claim 2, wherein the method further includes: when the sensory input information includes the gesture information, collecting the user’s sensory input information: identifying the gesture shape information corresponding to number. The user interface control method according to claim 1, wherein, after selecting the first functional area according to the eye movement information, it further includes: selecting the first functional area to indicate that it has been selected, so The selection prompt includes any one or combination of the following: highlight prompt, flash prompt, sound prompt or vibration prompt. An electronic device, which is improved in that it includes an eye tracker, a gesture recognition module, a voice recognition module, a display screen, and a processor; the processor is used to execute the user interface as described in any one of claims 1 to 8 Control method; the eye tracker is used to acquire eye movement information; the gesture recognition module is used to recognize gesture information; the voice recognition module is used to recognize voice information; the display screen is used to display user interface. A user interface is applied to a display screen, and the display screen is connected to an eye tracking module through communication. The improvement is that the user interface includes the electronic device as described in claim 9 when executing the user interface control method. The displayed user interface.

Images