CN113867851A

CN113867851A - Electronic equipment operation guide information recording method, electronic equipment operation guide information acquisition method and terminal equipment

Info

Publication number: CN113867851A
Application number: CN202010618162.7A
Authority: CN
Inventors: 刘超; 董金光
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2020-06-30
Filing date: 2020-06-30
Publication date: 2021-12-31
Also published as: CN116414500A

Abstract

The application is applicable to the technical field of terminals and provides an electronic equipment operation guide information recording method, an electronic equipment operation guide information acquisition method and terminal equipment. The method for recording the operation guide information of the electronic equipment comprises the following steps: the method comprises the steps that a first terminal device responds to a received electronic device type identifier and simulates an operation interface and/or physical hardware of a first electronic device corresponding to the electronic device type identifier; the first terminal equipment receives operation information based on the operation interface and/or the physical hardware; the first terminal equipment acquires operation element information generated according to the operation information; and the first terminal equipment records the operation element information to generate operation guide information, wherein the operation guide information comprises an electronic equipment type identifier. The method enables the first terminal device to acquire the operation element information of the first electronic device without the first electronic device.

Description

Electronic equipment operation guide information recording method, electronic equipment operation guide information acquisition method and terminal equipment

Technical Field

The present application relates to the field of terminal technologies, and in particular, to a method for recording and acquiring operation guidance information of an electronic device, and a terminal device.

Background

At present, with the types of electronic devices increasing, the operation of users on the electronic devices becomes increasingly complex, and especially for users who are not good at operating the electronic devices, how to simply and conveniently operate the electronic devices becomes a difficult problem.

Currently, an operation interface of a certain electronic device may be displayed on a terminal so that a user may perform an operation on the operation interface. Then, the terminal sends the operation of the user to the electronic equipment for execution so as to generate operation guide information. However, the above method requires the terminal to establish a communication connection with the electronic device, and how to record the operation guidance information for the electronic device that cannot establish a communication connection with the terminal device is an urgent problem to be solved.

Disclosure of Invention

The application provides an electronic equipment operation guide information recording method, an electronic equipment operation guide information acquisition method and a terminal device, which are used for realizing that the terminal device can acquire operation element information of the electronic equipment without the electronic equipment.

In order to achieve the purpose, the technical scheme is as follows:

in a first aspect, a method for recording operation guidance information of an electronic device is provided, where the method includes: the method comprises the steps that a first terminal device responds to a received electronic device type identifier and simulates an operation interface and/or physical hardware of a first electronic device corresponding to the electronic device type identifier; the first terminal equipment receives operation information based on the operation interface and/or the physical hardware; the first terminal equipment acquires operation element information generated according to the operation information; and the first terminal equipment records the operation element information to generate operation guide information, wherein the operation guide information comprises an electronic equipment type identifier.

In the embodiment of the application, the first terminal device simulates an operation interface and/or physical hardware of the electronic device. And then the first terminal equipment receives operation information based on the operation interface and/or the physical hardware and acquires operation element information generated according to the operation information. And then, the first terminal equipment records the operation element information to generate operation guide information. In the embodiment of the application, the first terminal device first simulates the operation interface and/or physical hardware of the first electronic device, since the function and attribute of the operation interface and/or physical hardware simulated by the first terminal device are the same as those of the operation interface and/or physical hardware of the first electronic device itself, therefore, the operation information based on the operation interface and/or the physical hardware acquired by the first terminal equipment is the same as the operation information generated by the first electronic equipment executing the same operation, the first terminal device generates the same operation element information based on the operation information as the operation element information generated by the first electronic device performing the operation information, therefore, the first terminal device can acquire the operation element information of the first electronic device without the first electronic device. In addition, in the embodiment of the application, the first electronic device does not need to share the own operation interface to the terminal device, so that the risk of revealing the privacy of the first electronic device user is avoided.

With reference to the first aspect, in some embodiments, simulating, by the first terminal device, an operation interface and/or physical hardware of an electronic device corresponding to the electronic device type identifier in response to the received electronic device type identifier includes: the first terminal equipment requests to establish communication connection with the first electronic equipment; and under the condition that the first terminal equipment does not successfully establish communication connection with the first electronic equipment, simulating the operation interface and/or the physical hardware of the first electronic equipment by the first terminal equipment.

With reference to the first aspect, in some embodiments, the simulating, by the first terminal device, the operation interface and/or the physical hardware of the electronic device includes: the first terminal device calls a virtual device model of the first electronic device, and the operation interface and/or the physical hardware of the first electronic device are/is simulated through the virtual device model.

For example, the first terminal device may simulate an operation interface and/or physical hardware of the first electronic device in an AR/VR manner, receive operation information input based on the operation interface and/or physical hardware in combination with a human body motion recognition technology, and acquire operation element information generated based on the operation information.

The virtual device model may include physical features, software features, operating component features of the first electronic device, and a 3D/2D device appearance/operation interface simulation model for presentation and operation in a virtual environment.

For example, the virtual device model may contain device IDs, device types, connection types, operations that may be recorded, hardware information, software information, 2D/3D model data, and the like. Wherein, the device ID is the identifier of the virtual device model; the device type is the device type of the first electronic device corresponding to the virtual device model; the connection type is a communication connection mode of the virtual device model, such as WIFI, Bluetooth, NFC, a local area network, a distributed device network and the like; the operation which can be recorded is used for limiting the operation which can be recorded in the operation of the virtual equipment model and can be used for preventing privacy information leakage; the hardware information represents the physical characteristics of the first electronic equipment corresponding to the virtual equipment model; the software information characterizes software features of an application program operable in the virtual device model; the 2D/3D model data is an appearance/operation interface simulation model and the like of the first electronic equipment corresponding to the virtual equipment model.

In combination with the first aspect, in some embodiments, the method comprises: the method comprises the steps that under the condition that the first terminal device and the first electronic device successfully establish communication connection, the first terminal device obtains operation element information of the first electronic device from the first electronic device.

With reference to the first aspect, in some embodiments, the obtaining, by the first terminal device, operation element information of the first electronic device from the first electronic device includes: the first terminal device sends an operation element information acquisition request to the first electronic device, where the operation element information acquisition request is used to request the first electronic device to acquire operation element information generated by the first electronic device in response to the received operation information, and sends the operation element information to the first terminal device.

With reference to the first aspect, in some embodiments, the electronic device type identifier is multiple, each electronic device type identifier corresponds to a first electronic device, each first electronic device corresponds to multiple sets of operating element information, and each set of operating element information corresponds to a generation time and an electronic device type identifier;

the method further comprises the following steps: the first terminal device sends a time synchronization request to each first electronic device corresponding to the electronic device type identifiers, wherein the time synchronization request is used for requesting the first electronic device to adjust the time of the first electronic device to be consistent with the time of the first terminal device;

the first terminal device records the operation element information to generate operation guide information, and the operation guide information includes: the first terminal equipment determines the operation element information corresponding to each first electronic equipment according to the electronic equipment type identification corresponding to each group of operation element information; and the first terminal equipment packs the operation element information corresponding to each first electronic equipment according to the sequence of the generation time of the operation element information to generate operation guide information corresponding to each first electronic equipment.

With reference to the first aspect, in some embodiments, the operation guidance information further includes text information, where the text information is: and the first electronic equipment converts the acquired audio information which is used for carrying out operation guidance together with the operation information into text information in a time range corresponding to the acquired operation information.

With reference to the first aspect, in some embodiments, the operation element information includes interface operation element information for an operation interface of the first electronic device, and/or hardware operation element information for physical hardware of the first electronic device; the interface operation element information comprises operation interface layout and operation element information corresponding to the state change of the operation interface.

For example, the operation page layout may include an application name, a page name, a button name, a location, and the like, and the state change of the operation interface may include a button click, an input box content change, a focus change, and the like, and the corresponding operation element information may be acquired based on the state change. For example, the terminal device may obtain the operation element information through a view service of a UI interface, and specifically, for a first terminal device of the android platform, the operation element information may be obtained through a local message monitoring function of an access availability service.

In combination with the first aspect, in some embodiments, the method further comprises: in a time range corresponding to the operation information acquired by the first terminal device, the first terminal device acquires audio information for performing operation guidance together with the operation information; and the first terminal equipment records the audio information in the operation guide information.

With reference to the first aspect, in some embodiments, the recording, by the first terminal device, the audio information in the operation guidance information includes: and the first terminal equipment converts the audio information into text information and records the text information in the operation guide information.

In order to avoid the risk of revealing the voiceprint information of the user, the first terminal device may convert the audio information into text information through an NLP technique and record the text information into operation guide information, and may convert the text information into audio information through a TTS technique in a subsequent process to play the audio information.

For example, the text information may include an electronic device type identifier and time information, where the electronic device type identifier is used to represent the first electronic device corresponding to the text information, and the time information is used to represent the time when the first terminal device acquires the text information.

In some embodiments, each piece of operation element information corresponds to a generation time, an electronic device type identifier, and an application identifier, and the method for recording operation guidance information of an electronic device may further include: the method comprises the steps that a first terminal device obtains an electronic device type identifier and an application program identifier corresponding to operation element information, establishes a link relation between the operation element information and the corresponding electronic device type identifier and application program identifier, and displays the electronic device type identifier and the application program identifier.

For example, the first terminal device may divide the operation element information according to the electronic device type identifier and the application identifier, divide the operation element information corresponding to each electronic device type identifier according to the application identifier, establish a link relationship between the operation element information and the electronic device type identifier and the application identifier, and display the electronic device type identifier and the application identifier in a list or a graph, so that the user may intuitively know the first electronic device and the application corresponding to the operation guidance information. Meanwhile, after receiving the operation information of the clicked electronic device type identifier and/or the application program identifier, the first terminal device may call corresponding operation element information based on the link relationship and play back the operation element information through the corresponding virtual device model, so that a user of the first terminal device can check whether the operation is complete. After receiving the input inspection completion information, the first terminal device may package the operation element information and issue the operation element information to the cloud for security inspection. The cloud end can perform security check on the operation element information related to the appointed APP in the operation element information and insert a security prompt into the operation element information for changing the sensitive setting of the system.

In a second aspect, a method for obtaining operation guidance information of an electronic device is provided, and the method includes: the second terminal equipment acquires operation guide information for carrying out operation guide on second electronic equipment, wherein the operation guide information comprises electronic equipment type identification and operation element information; the second terminal equipment simulates an operation interface and/or physical hardware of the second electronic equipment corresponding to the electronic equipment type identifier; and the second terminal equipment responds to the operation element information based on the operation interface and/or the physical hardware to generate prompt information for carrying out operation guidance on the second electronic equipment.

In the method for recording the operation guidance information of the electronic device, after the second terminal device acquires the operation guidance information for performing operation guidance on the second electronic device, the operation interface and/or the physical hardware of the second electronic device corresponding to the type identifier of the electronic device is simulated, and the operation guidance information is responded based on the operation interface and/or the physical hardware, because the function and the attribute of the operation interface and/or the physical hardware simulated by the second terminal device are the same as those of the operation interface and/or the physical hardware of the second electronic device, the operation information acquired by the second terminal device based on the operation interface and/or the physical hardware is the same as the operation information generated by the second electronic device based on the operation element information generated according to the operation information, and the operation element information generated by the second terminal device based on the operation element information generated according to the operation information is the same as the operation element information generated by the second electronic device executing the operation information The information is the same, so that the operation guidance effect of the second terminal device responding to the operation guidance information through the simulated operation interface and/or the physical hardware is the same as the operation guidance effect of the second electronic device responding to the operation guidance information, so that the second terminal device can respond to the operation guidance information without the second electronic device, and the user can be guided to operate the electronic device.

With reference to the second aspect, in some embodiments, the simulating, by the second terminal device, an operation interface and/or physical hardware of the second electronic device corresponding to the electronic device type identifier includes: under the condition that the second electronic equipment is electronic equipment without an operation guide information response unit, the second terminal equipment simulates the operation interface and/or the physical hardware of the second electronic equipment; wherein the operation guidance information response unit comprises a display unit and/or an audio playing unit.

In some embodiments, the method for recording the operation guidance information of the electronic device may further include: and the second terminal equipment determines whether the second electronic equipment corresponding to the electronic equipment type identification has an operation guide information response unit or not based on the electronic equipment type identification.

With reference to the second aspect, in some embodiments, the simulating, by the second terminal device, the operation interface and/or the physical hardware of the second electronic device includes: and the second terminal equipment calls a virtual equipment model of the second electronic equipment, and simulates the operation interface and/or the physical hardware of the second electronic equipment through the virtual equipment model.

The virtual device model of the second electronic device may be stored in the second terminal device, or may be stored in the cloud server, which is not limited in this embodiment of the present application. The virtual device model may comprise physical features, software features, operation component features of the second electronic device, and 3D/2D device appearance/operation interface simulation model for presentation and operation in a virtual environment

With reference to the second aspect, in some embodiments, the electronic device type identifier is multiple, each electronic device type identifier corresponds to a second electronic device, each second electronic device corresponds to an operation guidance information, each operation guidance information includes multiple sets of operation element information, and each set of operation element information corresponds to a generation time;

the responding to the operation guiding information based on the operation interface and/or the physical hardware comprises the following steps: the second terminal equipment analyzes the operation guide information corresponding to each second electronic equipment in sequence to obtain multiple groups of operation element information corresponding to each second electronic equipment; and the second terminal equipment responds to each group of operation element information sequentially through an operation interface and/or physical hardware of the second electronic equipment corresponding to each group of operation element information according to the sequence of the generation time corresponding to each group of operation element information.

In some embodiments, the operation element information includes interface operation element information for an operation interface of the second electronic device and/or hardware operation element information for physical hardware of the second electronic device; the interface operation element information comprises operation interface layout and operation element information corresponding to the state change of the operation interface.

The second terminal device responds to the operation guidance information based on the operation interface and/or the physical hardware, and specifically may be: and the second terminal equipment determines the position of the operation element information in the operation interface according to the layout of the operation interface in the operation element information, and displays the prompt information drawn based on the operation element information at the position.

For example, when a display interface needing to be operated exists in operation element information corresponding to a certain operation guidance step in the operation guidance information, the second terminal device may extract key element information in the corresponding operation element information and determine a position of the key element information in the operation interface, and display prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in a highlight frame or an arrow guide form; for the case that text information exists, the terminal device may convert the text information into audio information for playing using TTS.

In combination with the second aspect, in some embodiments, the method comprises: and if the second electronic equipment is the electronic equipment with the operation guide information response unit, the second terminal equipment sends the operation guide information to the second electronic equipment.

With reference to the second aspect, in some embodiments, the operation guidance information includes interface operation element information for a user interface of the second electronic device, hardware operation element information for physical hardware of the second electronic device, and text information converted from audio information for operation guidance; the interface operation element information is used for indicating a display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used for indicating an audio playing unit of the second electronic device to play the hardware operation element information, and the text information is used for indicating the audio playing unit of the second electronic device to convert the text information into audio information for playing.

In some embodiments, the method for recording the operation guidance information of the electronic device may further include: the second terminal equipment responds to the received operation instruction and sends the operation instruction to the corresponding second electronic equipment so that the second electronic equipment can execute the operation instruction; the operation instruction may be an operation instruction based on the operation guidance information received by the second terminal device after the second terminal device responds to the operation guidance information based on the operation interface and/or the physical hardware.

It can be understood that, after the second terminal device responds to the operation guidance information based on the operation interface and/or the physical hardware, the user may input an operation instruction for operating the second electronic device on the operation interface and/or the physical hardware simulated by the second terminal device, and the second terminal device sends the operation instruction to the corresponding second electronic device, so that the second electronic device can be operated by the second terminal device without the user going to the second electronic device to perform an operation.

For example, the second terminal device may simulate an operation interface and/or physical hardware of the second electronic device in an AR/VR manner, acquire an operation instruction through a human motion recognition technology, and send the operation instruction to the corresponding second electronic device, so that the second electronic device executes the operation instruction.

In a third aspect, a method for recording operation guidance information of an electronic device is provided, the method including: the method comprises the steps that first electronic equipment receives an operation element information acquisition request sent by first terminal equipment; the first electronic equipment responds to the operation element information acquisition request, receives input operation information and acquires operation element information generated based on the operation information; and the first electronic equipment sends the operation element information to the first terminal equipment.

According to the method for recording the operation guide information of the electronic equipment, after receiving an operation element information acquisition request sent by first terminal equipment, the first electronic equipment responds to the operation element information acquisition request, receives input operation information, acquires operation element information generated based on the operation information, and sends the operation element information to the first terminal equipment, so that the first terminal equipment records the operation element information to generate the operation guide information corresponding to the first electronic equipment, and because the operation element information does not relate to content which may reveal the user privacy of the electronic equipment compared with an operation interface image in the traditional technology, the risk of revealing the user privacy in the operation guide process of the electronic equipment can be reduced; the operation guide element information does not relate to the size of the operation interface, so that the method can be better suitable for electronic equipment with different operation interface sizes compared with an operation interface image in the traditional technology; and the space required by the operation element information is smaller than that of the operation interface image, so that the requirement on the network by the first electronic equipment in the process of transmitting the operation element information to the first terminal equipment is lower, and the time delay of data transmission can be reduced.

With reference to the third aspect, in some embodiments, the first electronic device receives a time synchronization request sent by the first terminal device, and adjusts, based on the time synchronization request, that a time of the first electronic device coincides with a time of the first terminal device; the sending, by the first electronic device, the operation element information to the first terminal device includes: and the first electronic equipment aggregates the operation element information generated by the first electronic equipment and then sends the operation element information to the first terminal equipment.

Illustratively, the operation element information may include a packet header, a node ID, a node type, an operation ID, a timestamp, operation data, and a packet trailer. The node ID represents the first electronic equipment corresponding to the operation element information; the node type represents the equipment type of the first electronic equipment; the operation ID is an operation identifier of the first electronic equipment or the virtual equipment model, wherein a data structure corresponding to an operation corresponding to the operation ID can be extracted from the first electronic equipment or the virtual equipment model according to a combination of the node type and the operation ID, and the combination can distinguish possible differences of definitions of the first electronic equipment with different operation interface sizes on the same operation; the time stamp is used to define the time at which the operation was performed.

In a fourth aspect, a method for directing information in response to operation of an electronic device is provided, the method comprising: the second electronic equipment receives operation guide information sent by the second terminal equipment, wherein the operation guide information comprises a plurality of groups of operation element information, and each group of operation element information corresponds to a generation time; and the second electronic equipment responds to each group of operation element information in sequence based on the generation time of each group of operation element information.

According to the method for recording the operation guide information of the electronic equipment, after the second electronic equipment receives the operation guide information sent by the second terminal equipment, the second electronic equipment responds to each group of operation element information in the operation guide information in sequence based on the generation time of each group of operation element information in the operation guide information, and as the operation element information is compared with an operation interface image in the traditional technology, the content which possibly reveals the user privacy of the electronic equipment is not involved, the risk of revealing the user privacy in the operation guide process of the electronic equipment can be reduced; and the operation guide element information does not relate to the size of the operation interface, so that the electronic equipment with different operation interface sizes can be better adapted to the operation interface image in the traditional technology.

With reference to the fourth aspect, in some embodiments, the operation guidance information includes one or more of interface operation element information for a user interface of the second electronic device, hardware operation element information for physical hardware of the second electronic device, and text information converted from audio information used for operation guidance; the second electronic device sequentially responds to each group of operation element information, and the method comprises the following steps: and the second electronic equipment displays the interface operation element information through a display unit according to the sequence of the generation time of each group of operation element information, and/or plays the hardware operation element information through an audio playing unit, and/or converts the text information into audio information through the audio playing unit for playing.

The electronic device may specifically respond to the operation guidance information by: and the second electronic equipment determines the position of the operation element information in the operation interface of the electronic equipment according to the operation interface layout in the operation element information, and displays the prompt information generated based on the operation element information at the position.

For example, when a display interface needing to be operated exists in operation element information corresponding to a certain operation guidance step in the operation guidance information, the second electronic device may extract key element information in the corresponding operation element information and determine a position of the key element information in the operation interface, and display prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in the form of a highlight frame or an arrow guide in the operation interface; for the case where text information is present, the second electronic device may convert the text information to audio information for playback using TTS.

In a fifth aspect, an apparatus for recording operation guidance information of an electronic device is provided, the apparatus comprising: the first simulation unit is used for responding to the received electronic equipment type identification and simulating an operation interface and/or physical hardware of the first electronic equipment corresponding to the electronic equipment type identification; the first receiving unit is used for receiving operation information based on the operation interface and/or the physical hardware; a first acquisition unit configured to acquire operation element information generated based on the operation information; and the recording unit is used for recording the operation element information and generating operation guide information.

In a sixth aspect, an apparatus for obtaining operation guidance information of an electronic device is provided, the apparatus comprising: the second acquisition unit is used for acquiring operation guide information for carrying out operation guide on second electronic equipment, and the operation guide information comprises an electronic equipment type identifier; the second simulation unit is used for simulating an operation interface and/or physical hardware of the second electronic equipment corresponding to the electronic equipment type identifier; and the first response unit is used for responding the operation guide information based on the operation interface and/or the physical hardware.

In a seventh aspect, an apparatus for recording operation guidance information of an electronic device is provided, the apparatus comprising: the second receiving unit is used for receiving an operation element information acquisition request sent by the first terminal equipment; a second response unit configured to receive, in response to the operation element information acquisition request, input operation information and acquire operation element information generated based on the operation information; and the sending unit is used for sending the operation element information to the first terminal equipment.

In an eighth aspect, there is provided an apparatus for directing information in response to operation of an electronic device, the apparatus comprising: the third receiving unit is used for receiving operation guide information sent by the second terminal device, wherein the operation guide information comprises a plurality of groups of operation element information, and each group of operation element information corresponds to a generation time; and the third response unit is used for sequentially responding to the groups of operation element information based on the generation time of each group of operation element information.

In a ninth aspect, embodiments of the present application provide a chip system, which may include a processor coupled with a memory, and the processor executes a computer program stored in the memory to implement the steps in the embodiments of the method described in the first aspect, or implement the steps in the embodiments of the method described in the second aspect, or implement the steps in the embodiments of the method described in the third aspect, or implement the steps in the embodiments of the method described in the fourth aspect. The chip system can be a single chip or a chip module consisting of a plurality of chips.

In a tenth aspect, embodiments of the present application provide a computer-readable storage medium, which stores a computer program that, when executed by a processor, implements the steps in the embodiments of the method according to any one of the first aspect, or implements the steps in the embodiments of the method according to the second aspect, or implements the steps in the embodiments of the method according to the third aspect, or implements the steps in the embodiments of the method according to the fourth aspect.

In an eleventh aspect, embodiments of the present application provide a computer program product, which, when run on a terminal device, causes an electronic device to perform the method of any one of the above first aspects, and/or implement the method of any one of the second aspects, and/or implement the method of any one of the third aspects, and/or implement the method of any one of the fourth aspects.

In a twelfth aspect, an embodiment of the present application provides a terminal device, where the terminal device includes: one or more processors, memory, and a display screen; the memory, the display screen, and the one or more processors, the memory to store computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by the one or more processors, cause the terminal device to perform the method of any of the first aspects above, or to implement the method of any of the second aspects above.

In a thirteenth aspect, an embodiment of the present application provides an electronic device, including: one or more processors, memory, and a display screen; the memory, the display screen, and the one or more processors, the memory to store computer program code, the computer program code comprising computer instructions; the computer instructions, when executed by the one or more processors, cause the terminal device to perform the method of any of the above third aspects, or to implement the method of any of the above fourth aspects.

It is to be understood that the electronic device operation guidance information recording apparatus of the fifth aspect to the eighth aspect, the chip system of the ninth aspect, the computer readable storage medium of the tenth aspect, the computer program product of the eleventh aspect, the terminal device of the twelfth aspect, and the electronic device of the thirteenth aspect provided above are all configured to perform the method provided in the first aspect to the fourth aspect. Therefore, the beneficial effects achieved by the method can refer to the beneficial effects in the corresponding method, and are not described herein again.

Drawings

Fig. 1 is a schematic view of an application scenario of a method for recording operation guidance information of an electronic device according to an embodiment of the present application;

fig. 2 is a schematic view of another application scenario of the method for recording operation guidance information of an electronic device according to an embodiment of the present application;

fig. 3 is a schematic diagram of a hardware architecture of a terminal device according to an embodiment of the present application;

fig. 4 is a block diagram of a software structure of a terminal device according to an embodiment of the present application;

fig. 5 is a schematic structural diagram of a terminal device according to an embodiment of the present application;

fig. 6 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application;

fig. 7 is a schematic view of an operation interface of a first terminal device according to an embodiment of the present application;

fig. 8 is a scene schematic diagram of an operation interface of a first terminal device according to an embodiment of the present application;

fig. 9 is a scene schematic diagram of an operation interface of a first terminal device according to an embodiment of the present application;

fig. 10 is a schematic flowchart illustrating a method for recording operation guidance information of an electronic device according to an embodiment of the present application;

fig. 11 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application;

fig. 12 is a flowchart illustrating a method for obtaining operation guidance information of an electronic device according to an embodiment of the present application

Fig. 13 is a schematic view of an operation interface of a second terminal device according to an embodiment of the present application;

fig. 14 is a scene schematic diagram of an operation interface of a second terminal device according to an embodiment of the present application;

fig. 15 is a scene schematic diagram of an operation interface of a second terminal device according to an embodiment of the present application;

fig. 16 is a schematic structural diagram of an apparatus for recording operation guidance information of an electronic device according to an embodiment of the present application

Fig. 17 is a schematic structural diagram of an apparatus for acquiring operation guidance information of an electronic device according to an embodiment of the present application;

fig. 18 is a schematic structural diagram of a terminal device according to an embodiment of the present application.

Detailed Description

The embodiments of the present application will be described below with reference to the drawings. The terminology used in the description of the embodiments herein is for the purpose of describing particular embodiments herein only and is not intended to be limiting of the application.

For the above problem, an operation interface of a certain electronic device may be displayed on the terminal device, so that the user may perform an operation on the operation interface. Then, the terminal device sends the operation of the user to the electronic device for execution so as to generate operation guide information.

For example, the terminal device B (assisted party) may request the terminal device a (assisting party) to assist the terminal device B in operating and guiding some programs or settings in the terminal device B through an in-person care Application (APP). Specifically, after the terminal device B establishes communication connection with the terminal device a through the familiarity care APP, the terminal device B may send an operation assistance request to the terminal device a through the APP, and after the terminal device a agrees with the operation assistance request of the terminal device B, the terminal device B sends an operation interface image of the terminal device B to the terminal device a. The user operates on the terminal device A according to the operation interface image, and then the terminal device A sends the operation information of the user aiming at the operation interface image to the terminal device B. And then the terminal device B can execute corresponding operation according to the operation information implemented by the user in the terminal device A, so that the terminal device B can be assisted by the terminal device A to realize the operation of the program or the setting in the terminal device B.

However, the above scheme is premised on: and establishing real-time synchronous communication connection between the terminal device A and the terminal device B. However, if the terminal device a and the terminal device B cannot establish a communication connection (for example, the terminal device B is not in an operating state, so that the terminal device a and the terminal device B cannot establish a communication connection), the terminal device a cannot generate operation guidance information of some programs or settings in the terminal device B according to the above scheme. In addition, since the operation interface image of the terminal device B is sent to the terminal device a, and the operation interface image may include the privacy information of the user of the terminal device B, there may be a risk that the privacy of the user is leaked, and the transmission of the operation interface image has a high requirement on the network connection quality.

Based on the above problem, in the embodiment of the present application, the terminal device simulates an operation interface and/or physical hardware of the electronic device. And then the terminal equipment receives operation information based on the operation interface and/or the physical hardware and acquires operation element information generated according to the operation information. And then, the terminal equipment records the operation element information to generate operation guide information. In the embodiment of the application, the terminal device simulates the operation interface and/or the physical hardware of the electronic device, and because the function and the attribute of the operation interface and/or the physical hardware simulated by the terminal device are the same as those of the operation interface and/or the physical hardware of the electronic device, the operation information obtained by the terminal device based on the operation interface and/or the physical hardware is the same as the operation information generated by the electronic device executing the same, and therefore the operation element information generated by the terminal device based on the operation information is the same as the operation element information generated by the electronic device executing the operation information, so that the terminal device can obtain the operation element information of the electronic device without the electronic device. In addition, the electronic equipment in the embodiment of the application does not need to share the own operation interface to the terminal equipment, so that the risk of revealing the privacy of the user of the electronic equipment is avoided.

As shown in fig. 1, fig. 1 shows a scenario to which an electronic device operation guidance information recording method provided in an embodiment of the present application is applicable, where the scenario includes: terminal equipment A, terminal equipment B, a plurality of electronic equipment a.

The terminal device a and the terminal device B have an asynchronous communication connection therebetween, for example, asynchronous message delivery can be performed between the terminal device a and the terminal device B.

When the user of the terminal device B is not good at operating the electronic device, the terminal device B may request the terminal device a to record the operation guidance information of the electronic device a, so as to assist the user of the terminal device B in operating the electronic device a.

The terminal device a may be a device having recording operation guidance information. The terminal device B may be a device that responds to the operation guidance information to display prompt information for operation guidance for the electronic device a.

The terminal device a may actively determine to record the operation guidance information, or the terminal device a may determine to record the operation guidance information based on the trigger of the terminal device B when the terminal device a successfully establishes a connection with the terminal device B, which is not limited in this embodiment of the present application.

The plurality of electronic devices a may include a Personal Computer (PC), a TV (television), a speaker, a router, a smart refrigerator, and other electronic devices.

In a possible implementation manner, the terminal device B may request, through the relevant application program, the terminal device a to assist in guiding the terminal device B to perform the operation of the electronic device a. And after receiving the request of the terminal device B, the terminal device A records the operation guide information of various electronic devices a according to the request. Then, the terminal device a delivers the operation guidance information of the various electronic devices a to the terminal device B.

The terminal device a may obtain the operation element information of each electronic device a in the plurality of electronic devices a through the terminal device a, and/or obtain the operation element information of the electronic device from the electronic device a, record the operation element information of each electronic device a to obtain the operation guidance information of each electronic device a, and send the operation guidance information of each electronic device a. For example, the terminal device a may transmit the operation guidance information of each electronic device a to the server or the terminal device B.

The terminal device B may acquire operation guidance information for performing operation guidance on each of the plurality of electronic devices a from the server or the terminal device a, and then generate prompt information for performing operation guidance on the electronic device a through the terminal device B and/or generate prompt information for performing operation guidance on the electronic device a through the corresponding electronic device a in response to the operation guidance information of each of the electronic devices a.

For example, the terminal device may be an electronic device such as a mobile phone, a tablet computer, a wearable device, an in-vehicle device, an Augmented Reality (AR)/Virtual Reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a Personal Digital Assistant (PDA), and the like, and the specific type of the electronic device is not limited in this embodiment.

As shown in fig. 2, fig. 2 shows another scenario in which the method for recording operation guidance information of an electronic device according to the embodiment of the present application is applied. This scenario differs from the scenario shown in fig. 1 in that: the scenario shown in fig. 2 includes a plurality of electronic devices b and a plurality of electronic devices a. The electronic device a or the electronic device b may be one of a plurality of electronic devices such as a PC, a TV, a speaker, a router, and an intelligent refrigerator. The type of the plurality of electronic devices a in which the partial electronic devices exist is the same as the type of the plurality of electronic devices. For example, the plurality of electronic devices a includes a PC, and the plurality of electronic devices b may include a PC and a TV. The plurality of electronic devices B may be electronic devices communicating with the terminal device a, and the plurality of electronic devices a may be electronic devices communicating with the terminal device B.

In the scenario shown in fig. 2, taking the second terminal device as an example of requesting the first terminal device to generate the operation guidance information of the electronic device a, if the first terminal device determines that the electronic device b communicating with the first terminal device is the same as the electronic device a in type, the first terminal device may simulate the operation interface and/or the physical hardware of the electronic device b, in other words, the first terminal device may use the simulated operation interface and/or the physical hardware of the electronic device b as the operation interface and/or the physical hardware of the electronic device a. In the scenario shown in fig. 1, the first terminal device simulates an operation interface and/or physical hardware of the electronic device a.

For example, fig. 3 is a schematic diagram of a hardware architecture of the terminal device 100 provided in the embodiment of the present application, and the structures of the terminal device a and the terminal device B may refer to the description of the terminal device 100, which is not described herein again. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a Universal Serial Bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a Subscriber Identity Module (SIM) card interface 195, and the like. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, and the like.

It is to be understood that the illustrated structure of the embodiment of the present invention does not specifically limit the terminal device 100. In other embodiments of the present application, terminal device 100 may include more or fewer components than shown, or some components may be combined, some components may be split, or a different arrangement of components. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

Processor 110 may include one or more processing units, such as: the processor 110 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a memory, a video codec, a Digital Signal Processor (DSP), a baseband processor, and/or a neural-Network Processing Unit (NPU), etc. The different processing units may be separate devices or may be integrated into one or more processors.

The controller may be a neural center and a command center of the terminal device 100, among others. The controller can generate an operation control signal according to the instruction operation code and the timing signal to complete the control of instruction fetching and instruction execution.

A memory may also be provided in processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may hold instructions or data that have just been used or recycled by the processor 110. If the processor 110 needs to reuse the instruction or data, it can be called directly from memory. Avoiding repeated accesses reduces the latency of the processor 110, thereby increasing the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (I2C) interface, an integrated circuit built-in audio (I2S) interface, a Pulse Code Modulation (PCM) interface, a universal asynchronous receiver/transmitter (UART) interface, a Mobile Industry Processor Interface (MIPI), a general-purpose input/output (GPIO) interface, a Subscriber Identity Module (SIM) interface, and/or a Universal Serial Bus (USB) interface, etc.

The I2C interface is a bi-directional synchronous serial bus that includes a serial data line (SDA) and a Serial Clock Line (SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces, respectively. For example: the processor 110 may be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through an I2C bus interface to implement the touch function of the terminal device 100.

The I2S interface may be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 via an I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 may communicate audio signals to the wireless communication module 160 via the I2S interface, enabling answering of calls via a bluetooth headset.

The PCM interface may also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled by a PCM bus interface. In some embodiments, the audio module 170 may also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to implement a function of answering a call through a bluetooth headset. Both the I2S interface and the PCM interface may be used for audio communication.

The UART interface is a universal serial data bus used for asynchronous communications. The bus may be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 with the wireless communication module 160. For example: the processor 110 communicates with a bluetooth module in the wireless communication module 160 through a UART interface to implement a bluetooth function. In some embodiments, the audio module 170 may transmit the audio signal to the wireless communication module 160 through a UART interface, so as to realize the function of playing music through a bluetooth headset.

MIPI interfaces may be used to connect processor 110 with peripheral devices such as display screen 194, camera 193, and the like. The MIPI interface includes a Camera Serial Interface (CSI), a Display Serial Interface (DSI), and the like. In some embodiments, processor 110 and camera 193 communicate through a CSI interface to implement the capture function of terminal device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the terminal device 100.

The GPIO interface may be configured by software. The GPIO interface may be configured as a control signal and may also be configured as a data signal. In some embodiments, a GPIO interface may be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, and the like. The GPIO interface may also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, and the like.

The USB interface 130 is an interface conforming to the USB standard specification, and may specifically be a Mini USB interface, a Micro USB interface, a USB Type C interface, or the like. The USB interface 130 may be used to connect a charger to charge the terminal device 100, and may also be used to transmit data between the terminal device 100 and a peripheral device. And the earphone can also be used for connecting an earphone and playing audio through the earphone. The interface may also be used to connect other electronic devices, such as AR devices and the like.

It should be understood that the interface connection relationship between the modules according to the embodiment of the present invention is only an exemplary illustration, and does not limit the structure of the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection manners or a combination of multiple interface connection manners in the above embodiments.

The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the terminal device 100. The charging management module 140 may also supply power to the electronic device through the power management module 141 while charging the battery 142.

The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charge management module 140 and provides power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 may also be used to monitor parameters such as battery capacity, battery cycle count, battery state of health (leakage, impedance), etc. In some other embodiments, the power management module 141 may also be disposed in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be disposed in the same device.

The wireless communication function of the terminal device 100 may be implemented by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, a modem processor, a baseband processor, and the like.

The antennas 1 and 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in terminal device 100 may be used to cover a single or multiple communication bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: the antenna 1 may be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 may provide a solution including 2G/3G/4G/5G wireless communication applied on the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a Low Noise Amplifier (LNA), and the like. The mobile communication module 150 may receive the electromagnetic wave from the antenna 1, filter, amplify, etc. the received electromagnetic wave, and transmit the electromagnetic wave to the modem processor for demodulation. The mobile communication module 150 may also amplify the signal modulated by the modem processor, and convert the signal into electromagnetic wave through the antenna 1 to radiate the electromagnetic wave. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 may be disposed in the same device as at least some of the modules of the processor 110.

The modem processor may include a modulator and a demodulator. The modulator is used for modulating a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used for demodulating the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then passes the demodulated low frequency baseband signal to a baseband processor for processing. The low frequency baseband signal is processed by the baseband processor and then transferred to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be provided in the same device as the mobile communication module 150 or other functional modules, independent of the processor 110.

The wireless communication module 160 may provide a solution for wireless communication applied to the terminal device 100, including Wireless Local Area Networks (WLANs) (e.g., wireless fidelity (Wi-Fi) networks), bluetooth (bluetooth, BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), and the like. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, performs frequency modulation and filtering processing on electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 may also receive a signal to be transmitted from the processor 110, perform frequency modulation and amplification on the signal, and convert the signal into electromagnetic waves through the antenna 2 to radiate the electromagnetic waves.

In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150 and the antenna 2 is coupled to the wireless communication module 160 so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), General Packet Radio Service (GPRS), code division multiple access (code division multiple access, CDMA), Wideband Code Division Multiple Access (WCDMA), time-division code division multiple access (time-division code division multiple access, TD-SCDMA), Long Term Evolution (LTE), LTE, BT, GNSS, WLAN, NFC, FM, and/or IR technologies, among others. GNSS may include Global Positioning System (GPS), global navigation satellite system (GLONASS), beidou satellite navigation system (BDS), quasi-zenith satellite system (QZSS), and/or Satellite Based Augmentation System (SBAS).

The terminal device 100 implements a display function by the GPU, the display screen 194, and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and an application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.

The display screen 194 is used to display images, video, and the like. The display screen 194 includes a display panel. The display panel may adopt a Liquid Crystal Display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (active-matrix organic light-emitting diode, AMOLED), a flexible light-emitting diode (FLED), a miniature, a Micro-oeld, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

The terminal device 100 may implement a shooting function through the ISP, the camera 193, the video codec, the GPU, the display screen 194, the application processor, and the like.

The ISP is used to process the data fed back by the camera 193. For example, when a photo is taken, the shutter is opened, light is transmitted to the camera photosensitive element through the lens, the optical signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to the ISP for processing and converting into an image visible to naked eyes. The ISP can also carry out algorithm optimization on the noise, brightness and skin color of the image. The ISP can also optimize parameters such as exposure, color temperature and the like of a shooting scene. In some embodiments, the ISP may be provided in camera 193.

The camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects the optical image to the photosensitive element. The photosensitive element may be a Charge Coupled Device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The light sensing element converts the optical signal into an electrical signal, which is then passed to the ISP where it is converted into a digital image signal. And the ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into image signal in standard RGB, YUV and other formats. In some embodiments, the terminal device 100 may include 1 or N cameras 193, N being a positive integer greater than 1.

The digital signal processor is used for processing digital signals, and can process digital image signals and other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform fourier transform or the like on the frequency point energy.

Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record video in a plurality of encoding formats, such as: moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, and the like.

The NPU is a neural-network (NN) computing processor that processes input information quickly by using a biological neural network structure, for example, by using a transfer mode between neurons of a human brain, and can also learn by itself continuously. The NPU can implement applications such as intelligent recognition of the terminal device 100, for example: image recognition, face recognition, speech recognition, text understanding, and the like.

The external memory interface 120 may be used to connect an external memory card, such as a Micro SD card, to extend the storage capability of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. For example, files such as music, video, etc. are saved in an external memory card.

The internal memory 121 may be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications of the terminal device 100 and data processing by executing instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required by at least one function, and the like. The storage data area may store data (such as audio data, a phonebook, etc.) created during use of the terminal device 100, and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device, a flash memory device, a universal flash memory (UFS), and the like.

The terminal device 100 may implement an audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playing, recording, etc.

The audio module 170 is used to convert digital audio information into an analog audio signal output and also to convert an analog audio input into a digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be disposed in the processor 110, or some functional modules of the audio module 170 may be disposed in the processor 110.

The speaker 170A, also called a "horn", is used to convert the audio electrical signal into an acoustic signal. The terminal device 100 can listen to music through the speaker 170A, or listen to a handsfree call.

The receiver 170B, also called "earpiece", is used to convert the electrical audio signal into an acoustic signal. When the terminal device 100 answers a call or voice information, it is possible to answer a voice by bringing the receiver 170B close to the human ear.

The microphone 170C, also referred to as a "microphone," is used to convert sound signals into electrical signals. When making a call or transmitting voice information, the user can input a voice signal to the microphone 170C by speaking the user's mouth near the microphone 170C. The terminal device 100 may be provided with at least one microphone 170C. In other embodiments, the terminal device 100 may be provided with two microphones 170C, which may implement a noise reduction function in addition to collecting sound signals. In other embodiments, the terminal device 100 may further include three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and implement directional recording functions.

The headphone interface 170D is used to connect a wired headphone. The headset interface 170D may be the USB interface 130, or may be a 3.5mm open mobile electronic device platform (OMTP) standard interface, a cellular telecommunications industry association (cellular telecommunications industry association of the USA, CTIA) standard interface.

The pressure sensor 180A is used for sensing a pressure signal, and converting the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A may be disposed on the display screen 194. The pressure sensor 180A can be of a wide variety, such as a resistive pressure sensor, an inductive pressure sensor, a capacitive pressure sensor, and the like. The capacitive pressure sensor may be a sensor comprising at least two parallel plates having an electrically conductive material. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure from the change in the capacitance. When a touch operation is applied to the display screen 194, the terminal device 100 detects the intensity of the touch operation from the pressure sensor 180A. The terminal device 100 may also calculate the touched position from the detection signal of the pressure sensor 180A. In some embodiments, the touch operations that are applied to the same touch position but different touch operation intensities may correspond to different operation instructions. For example: and when the touch operation with the touch operation intensity smaller than the first pressure threshold value acts on the short message application icon, executing an instruction for viewing the short message. And when the touch operation with the touch operation intensity larger than or equal to the first pressure threshold value acts on the short message application icon, executing an instruction of newly building the short message.

The gyro sensor 180B may be used to determine the motion attitude of the terminal device 100. In some embodiments, the angular velocity of terminal device 100 about three axes (i.e., x, y, and z axes) may be determined by gyroscope sensor 180B. The gyro sensor 180B may be used for photographing anti-shake. Illustratively, when the shutter is pressed, the gyro sensor 180B detects the shake angle of the terminal device 100, calculates the distance to be compensated for by the lens module according to the shake angle, and allows the lens to counteract the shake of the terminal device 100 through a reverse movement, thereby achieving anti-shake. The gyroscope sensor 180B may also be used for navigation, somatosensory gaming scenes.

The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates an altitude from the barometric pressure measured by the barometric pressure sensor 180C, and assists in positioning and navigation.

The magnetic sensor 180D includes a hall sensor. The terminal device 100 may detect the opening and closing of the flip holster using the magnetic sensor 180D. In some embodiments, when the terminal device 100 is a folder, the terminal device 100 may detect the opening and closing of the folder according to the magnetic sensor 180D. And then according to the opening and closing state of the leather sheath or the opening and closing state of the flip cover, the automatic unlocking of the flip cover is set.

The acceleration sensor 180E can detect the magnitude of acceleration of the terminal device 100 in various directions (generally, three axes). The magnitude and direction of gravity can be detected when the terminal device 100 is stationary. The method can also be used for recognizing the posture of the electronic equipment, and is applied to horizontal and vertical screen switching, pedometers and other applications.

A distance sensor 180F for measuring a distance. The terminal device 100 may measure the distance by infrared or laser. In some embodiments, shooting a scene, the terminal device 100 may range using the distance sensor 180F to achieve fast focus.

The proximity light sensor 180G may include, for example, a Light Emitting Diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light to the outside through the light emitting diode. The terminal device 100 detects infrared reflected light from a nearby object using a photodiode. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can utilize the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear for talking, so as to automatically turn off the screen to achieve the purpose of saving power. The proximity light sensor 180G may also be used in a holster mode, a pocket mode automatically unlocks and locks the screen.

The ambient light sensor 180L is used to sense the ambient light level. The terminal device 100 may adaptively adjust the brightness of the display screen 194 according to the perceived ambient light level. The ambient light sensor 180L may also be used to automatically adjust the white balance when taking a picture. The ambient light sensor 180L may also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket, in order to prevent accidental touches.

The fingerprint sensor 180H is used to collect a fingerprint. The terminal device 100 can utilize the collected fingerprint characteristics to realize fingerprint unlocking, access to an application lock, fingerprint photographing, fingerprint incoming call answering and the like.

The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 executes a temperature processing policy using the temperature detected by the temperature sensor 180J. For example, when the temperature reported by the temperature sensor 180J exceeds the threshold, the terminal device 100 performs a reduction in performance of a processor located near the temperature sensor 180J, so as to reduce power consumption and implement thermal protection. In other embodiments, the terminal device 100 heats the battery 142 when the temperature is below another threshold to avoid the terminal device 100 being abnormally shut down due to low temperature. In other embodiments, when the temperature is lower than a further threshold, the terminal device 100 performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown due to low temperature.

The touch sensor 180K is also referred to as a "touch panel". The touch sensor 180K may be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, which is also called a "touch screen". The touch sensor 180K is used to detect a touch operation applied thereto or nearby. The touch sensor can communicate the detected touch operation to the application processor to determine the touch event type. Visual output associated with the touch operation may be provided through the display screen 194. In other embodiments, the touch sensor 180K may be disposed on the surface of the terminal device 100, different from the position of the display screen 194.

The bone conduction sensor 180M may acquire a vibration signal. In some embodiments, the bone conduction sensor 180M may acquire a vibration signal of the human vocal part vibrating the bone mass. The bone conduction sensor 180M may also contact the human pulse to receive the blood pressure pulsation signal. In some embodiments, the bone conduction sensor 180M may also be disposed in a headset, integrated into a bone conduction headset. The audio module 170 may analyze a voice signal based on the vibration signal of the bone mass vibrated by the sound part acquired by the bone conduction sensor 180M, so as to implement a voice function. The application processor can analyze heart rate information based on the blood pressure beating signals acquired by the bone conduction sensor 180M, and the heart rate detection function is realized.

The keys 190 include a power-on key, a volume key, and the like. The keys 190 may be mechanical keys. Or may be touch keys. The terminal device 100 may receive a key input, and generate a key signal input related to user setting and function control of the terminal device 100.

The motor 191 may generate a vibration cue. The motor 191 may be used for incoming call vibration cues, as well as for touch vibration feedback. For example, touch operations applied to different applications (e.g., photographing, audio playing, etc.) may correspond to different vibration feedback effects. The motor 191 may also respond to different vibration feedback effects for touch operations applied to different areas of the display screen 194. Different application scenes (such as time reminding, receiving information, alarm clock, game and the like) can also correspond to different vibration feedback effects. The touch vibration feedback effect may also support customization.

Indicator 192 may be an indicator light that may be used to indicate a state of charge, a change in charge, or a message, missed call, notification, etc.

The SIM card interface 195 is used to connect a SIM card. The SIM card can be brought into and out of contact with the terminal device 100 by being inserted into the SIM card interface 195 or being pulled out of the SIM card interface 195. The terminal device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 may support a Nano SIM card, a Micro SIM card, a SIM card, etc. The same SIM card interface 195 can be inserted with multiple cards at the same time. The types of the plurality of cards may be the same or different. The SIM card interface 195 may also be compatible with different types of SIM cards. The SIM card interface 195 may also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as communication and data communication. In some embodiments, the terminal device 100 employs eSIM, namely: an embedded SIM card. The eSIM card may be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

The software system of the terminal device 100 may adopt a hierarchical architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes an Android system with a layered architecture as an example, and exemplarily illustrates a software structure of the terminal device 100.

Fig. 4 is a block diagram of a software configuration of the terminal device 100 according to the embodiment of the present application.

The layered architecture divides the software into several layers, each layer having a clear role and division of labor. The layers communicate with each other through a software interface. In some embodiments, the Android system is divided into four layers, an application layer, an application framework layer, an Android runtime (Android runtime) and system library, and a kernel layer from top to bottom.

The application layer may include a series of application packages.

As shown in fig. 4, the application package may include applications such as camera, gallery, calendar, phone call, map, navigation, WLAN, bluetooth, music, video, short message, etc.

The application framework layer provides an Application Programming Interface (API) and a programming framework for the application program of the application layer. The application framework layer includes a number of predefined functions.

As shown in FIG. 4, the application framework layers may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and the like.

The window manager is used for managing window programs. The window manager can obtain the size of the display screen, judge whether a status bar exists, lock the screen, intercept the screen and the like.

The content provider is used to store and retrieve data and make it accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

The view system includes visual controls such as controls to display text, controls to display pictures, and the like. The view system may be used to build applications. The display interface may be composed of one or more views. For example, the display interface including the short message notification icon may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide the communication function of the terminal device 100. Such as management of call status (including on, off, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and the like.

The notification manager enables the application to display notification information in the status bar, can be used to convey notification-type messages, can disappear automatically after a short dwell, and does not require user interaction. Such as a notification manager used to inform download completion, message alerts, etc. The notification manager may also be a notification that appears in the form of a chart or scroll bar text at the top status bar of the system, such as a notification of a background running application, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, sounding a prompt tone, vibrating the electronic device, flashing an indicator light, etc.

The Android Runtime comprises a core library and a virtual machine. The Android runtime is responsible for scheduling and managing an Android system.

The core library comprises two parts: one part is a function which needs to be called by java language, and the other part is a core library of android.

The application layer and the application framework layer run in a virtual machine. And executing java files of the application program layer and the application program framework layer into a binary file by the virtual machine. The virtual machine is used for performing the functions of object life cycle management, stack management, thread management, safety and exception management, garbage collection and the like.

The system library may include a plurality of functional modules. For example: surface managers (surface managers), Media Libraries (Media Libraries), three-dimensional graphics processing Libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), and the like.

The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

The media library supports a variety of commonly used audio, video format playback and recording, and still image files, among others. The media library may support a variety of audio-video encoding formats, such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG, and the like.

The three-dimensional graphic processing library is used for realizing three-dimensional graphic drawing, image rendering, synthesis, layer processing and the like.

The 2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is a layer between hardware and software. The inner core layer at least comprises a display driver, a camera driver, an audio driver and a sensor driver.

The following describes exemplary workflow of the software and hardware of the terminal device 100 in connection with capturing a photo scene.

When the touch sensor 180K receives a touch operation, a corresponding hardware interrupt is issued to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, a time stamp of the touch operation, and other information). The raw input events are stored at the kernel layer. And the application program framework layer acquires the original input event from the kernel layer and identifies the control corresponding to the input event. Taking the touch operation as a touch click operation, and taking a control corresponding to the click operation as a control of a camera application icon as an example, the camera application calls an interface of an application framework layer, starts the camera application, further starts a camera drive by calling a kernel layer, and captures a still image or a video through the camera 193.

As shown in fig. 5, fig. 5 provides a schematic diagram of an architecture of a terminal device, and the architectures of the terminal device a and the terminal device B may both refer to the architecture shown in fig. 4, and the terminal device may include a device management module, a connection management module, a data processing module, and an operation playback module. The device management module may correspond to the processor 110, the internal memory 121, the external memory interface 120, and the USB interface 130 in fig. 3, the connection management module may correspond to the processor 110 and the wireless communication module 160 in fig. 3, the data processing module may correspond to the processor 110 in fig. 3, and the operation playback module may correspond to the processor 110, the display screen, and the speaker 170A in fig. 3.

Specifically, the connection management module includes a local device connection management unit and a virtual device connection management unit, where the virtual device connection management unit is used to manage connection between the remote device and the virtual device model, and the remote device data is mapped to the current main device executing the recording operation in a device model manner.

The data processing module comprises a local operation monitoring unit, a virtual equipment monitoring unit and an operation data packaging unit and is used for recording operation events in the operation recording process of the electronic equipment and providing operations of data aggregation processing, authentication packaging and the like of the operation events.

The operation playback module comprises a local playback unit and a receiving end playback unit and is used for operation of relevant operations such as process check after recording, prompt analysis and distribution of an execution end and the like.

Here, the local playback unit may not be provided in the terminal device B.

The device management module comprises a physical device model management unit, a virtual device model management unit and a virtual device simulation component, wherein the physical device model and the virtual device model comprise physical characteristics of an actual product, software version characteristics, recordable operation component characteristics and a 3D/2D device appearance/operation interface simulation model which is possibly contained and is provided for showing and operating in a virtual environment.

The following describes an operation guidance method for an electronic device provided in an embodiment of the present application.

Referring to fig. 6, fig. 6 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application, where the method includes:

step S101, the first terminal device responds to the received electronic device type identification and simulates an operation interface and/or physical hardware of the first electronic device corresponding to the electronic device type identification.

For example, the first terminal device may be terminal device a shown in fig. 1 or fig. 2. For example, the first electronic device may be the electronic device a in fig. 1 or fig. 2. The physical hardware of the first electronic device may refer to physical hardware inherent to the first electronic device, such as a switch button, volume +, volume down button, etc. of a sound box, an antenna for routing, and a switch, etc.

The number of the electronic device type identifiers can be one or more.

It should be understood that, in the embodiment of the present application, before step S101, the method provided in the embodiment of the present application further includes: the first terminal equipment acquires one or more electronic equipment type identifications.

In a possible implementation manner, in this embodiment of the application, the obtaining, by the first terminal device, one or more electronic device type identifiers may be implemented by: the first terminal device receives one or more electronic device type identifications from a second terminal device (e.g., terminal device B shown in fig. 1 or fig. 2). For example, terminal device B may send a request message to terminal device a through the relevant application. The request message is used to request the terminal device a to record the operation guidance information. The operation guidance information is used for guiding the user of the terminal device B how to operate the one or more electronic devices, and the request may carry the type identifier of the one or more electronic devices.

In another possible implementation manner, in this embodiment of the application, the obtaining, by the first terminal device, one or more electronic device type identifiers may be implemented by: the first terminal device receives a first operation input by a user, and the first operation is used for determining the one or more electronic device type identifications.

Referring to (a) in fig. 7, the first terminal device displays at least the operation guidance APP200 as shown in (a) in fig. 7 through the display interface. After the first terminal device detects that the user clicks the operation guidance APP200 shown in (a) in fig. 7, the first terminal device opens the operation guidance APP200, and displays an interface shown in (b) in fig. 7 or an interface shown in (d) in fig. 7. As shown in (b) of fig. 7, the first terminal device displays information of one or more electronic devices for selection by the user. As shown in (c) in fig. 7, the user may click controls corresponding to some or all of the electronic devices in the display interface of the first terminal device. When the first terminal device detects that the user clicks the "confirm" control shown in (c) of fig. 7, and the first terminal device determines that the "confirm" control is triggered, the first terminal device may determine the selected "personal computer", "speaker a", "television", "router", "refrigerator", and the like in (c) of fig. 7 as the one or more electronic device type identifiers selected by the user. Thereafter, the first terminal device simulates the operation interface and/or physical hardware of the electronic device selected by the user, as shown in (f) of fig. 7, the first terminal device may simulate the operation interface and/or physical hardware of one or more electronic devices selected by the user clicking in the first operation interface region 201 of the first terminal device.

As shown in (d) of fig. 7, the first terminal device identifies the type of the one or more electronic devices carried in the request sent by the second terminal device. And the first terminal equipment displays the control of the electronic equipment corresponding to the type identification of the one or more electronic equipment. As shown in fig. 7 (e), after confirming that the second terminal device requests the assistance operation to guide the electronic device, the user clicks the "confirm" control. The first terminal device simulates the operation interface and/or physical hardware of the one or more electronic devices in response to the one or more electronic devices confirmed by the user, as shown in (f) of fig. 7, and the first terminal device may simulate the operation interface and/or physical hardware of the one or more electronic devices confirmed by the user in the first operation interface area 201 of the first terminal device.

For example, taking the first electronic device as a personal computer as an example, as shown in (a) in fig. 8, the first terminal device displays an operation interface 201 of the personal computer simulated by the first terminal device. The user can click on each control in the operation interface 201 of the simulated personal computer, for example, the user clicks on a control of a 'control panel'. After responding to the click operation of the control panel, the first terminal device simulates an operation interface of a personal computer shown in (b) of fig. 8. The user can click on each control in the operation interface of the personal computer as shown in (b) in fig. 8, for example, the user clicks on a "program and function" control. The first terminal device simulates an operation interface of a personal computer shown in (c) in fig. 8 after responding to the click operation on the "program and function" control, and the user can click and operate each control in the operation interface of the personal computer shown in (c) in fig. 8. For example, the user clicks on the "Foxmail" control. The first terminal device simulates an operation interface of a personal computer as shown in (d) in fig. 8 in response to a click operation on the "Foxmail" control, and a user can click on the "uninstall" control in the operation interface of the personal computer as shown in (d) in fig. 8.

For example, taking the first electronic device as a router as an example, as shown in fig. 9, the first terminal device displays physical hardware of the router simulated by the first terminal device, and a user may operate some or some of the physical hardware of the router on the physical hardware of the simulated router, for example, the user may adjust an angle of an antenna of the router simulated by the first terminal device.

And S102, the first terminal equipment receives operation information based on the operation interface and/or the physical hardware.

After the first terminal device simulates an operation interface and/or physical hardware of the electronic device, a user can operate on the operation interface and/or the physical hardware. For example, the user performs operations such as clicking and sliding on the operation interface, performs operations such as adjusting and rotating on the physical hardware, and the corresponding operation information may include operation contents, position information, and the like.

For example, taking the first electronic device as an audio amplifier as an example, the physical hardware of the first electronic device may be a switch button, a volume +, a volume down button, and the like of the audio amplifier. Taking the first electronic device as an example of a router, the physical hardware of the first electronic device may be physical hardware such as an antenna, a switch button, a network cable interface, and the like of the router, the button or the antenna may be adjusted by a user operating the simulated physical hardware in the first terminal device, and the first terminal device may obtain operation information corresponding to the physical hardware by detecting an operation performed by the user on the simulated operation interface and/or the physical hardware.

Step S103, the first terminal device obtains the operation element information generated according to the operation information.

In an application scenario, for the operation interface simulated in the first operation interface region 201 of the first terminal device, the first terminal device receives operation information input based on the operation interface, and acquires operation element information generated according to the operation information. For example, the operation may be a touch operation or a touch operation set for an application on the operation interface, and the operation element information may include interface operation element information for the operation interface and/or hardware operation element information for physical hardware of the electronic device, and the interface operation element information may include an operation interface layout and operation element information corresponding to a state change of the operation interface.

For example, the operation page layout may include an application name, a page name, a button name, a location, and the like, and the state change of the operation interface may include a button click, an input box content change, a focus change, and the like, and the corresponding operation element information may be acquired based on the state change. For example, the first terminal device may obtain the operation element information through a view service of a User Interface (UI), and specifically, for a first terminal device of an android platform, the operation element information may be obtained through a local message monitoring function of an auxiliary service (access service).

In an application scenario, for the physical hardware simulated in the first operation interface region 201 of the first terminal device, the first terminal device receives operation information input based on the physical hardware, and acquires operation element information corresponding to the operation information in a system message generated by monitoring the operation. For example, the physical hardware may be physical hardware such as a button of a sound box, an antenna of a router, and the like, the operation of the physical hardware simulated by the user on the first terminal device may adjust the button or the antenna, and the first terminal device may monitor a system message generated by the operation, and acquire hardware operation element information corresponding to the physical hardware from the system message.

For example, the first terminal device may simulate an operation interface and/or physical hardware of the first electronic device in an AR (Augmented Reality)/VR (Virtual Reality) manner, receive operation information input based on the operation interface and/or physical hardware in combination with a human body motion recognition technology, and acquire operation element information generated according to the operation information.

And step S104, the first terminal equipment records the operation element information to generate operation guide information, wherein the operation guide information comprises the type identification of the electronic equipment.

The first terminal device may record operation element information corresponding to the electronic device type identifier, and generate operation guidance information corresponding to the electronic device type identifier, where the operation guidance information includes the corresponding electronic device type identifier. When the electronic equipment type identifiers are multiple, the first terminal equipment records the operation element information corresponding to each electronic equipment type identifier, and generates operation guide information corresponding to each electronic equipment type identifier.

In the method for recording the operation guide information of the electronic equipment, the first terminal equipment simulates the operation interface and/or the physical hardware of the first electronic equipment, since the function and attribute of the operation interface and/or physical hardware simulated by the first terminal device are the same as those of the operation interface and/or physical hardware of the first electronic device itself, therefore, the operation information based on the operation interface and/or the physical hardware acquired by the first terminal equipment is the same as the operation information generated by the first electronic equipment executing the same operation, the first terminal device generates the same operation element information based on the operation information as the operation element information generated by the first electronic device performing the operation information, therefore, the first terminal device can acquire the operation element information of the first electronic device without the first electronic device. In addition, in the embodiment of the application, the first electronic device does not need to share the own operation interface to the terminal device, so that the risk of revealing the privacy of the first electronic device user is avoided.

For example, the electronic device type identifier may include one or more of an electronic device type identifier corresponding to a PC, an electronic device type identifier corresponding to a TV, an electronic device type identifier corresponding to a smart refrigerator, an electronic device type identifier corresponding to a speaker, and an electronic device type identifier corresponding to a route.

For the case of multiple electronic device type identifiers, the simulating, by the first terminal device, an operation interface and/or physical hardware of the first electronic device corresponding to the electronic device type identifier may specifically be:

the first terminal device sequentially simulates the operation interfaces and/or physical hardware of the first electronic devices corresponding to the type identifiers of the electronic devices according to a preset sequence, so that corresponding operation element information can be obtained respectively based on the operation information corresponding to each first electronic device, and finally, the operation element information corresponding to each first electronic device is recorded respectively, and operation guide information corresponding to each first electronic device is generated.

For example, each piece of operation element information may correspond to a generation time and an electronic device type identifier, the generation time may be based on the time of the first terminal device, and based on this, the first terminal device may determine the operation element information corresponding to each first electronic device according to the generation time and the electronic device type identifier corresponding to the operation element information; and the first terminal equipment packages the operation element information corresponding to each electronic equipment and generates operation guide information corresponding to each electronic equipment.

In the embodiment of the application, the terminal device simulates an operation interface and/or physical hardware of the electronic device. And then the terminal equipment receives operation information based on the operation interface and/or the physical hardware and acquires operation element information generated according to the operation information. And then, the terminal equipment records the operation element information to generate operation guide information. In the embodiment of the application, the terminal device simulates the operation interface and/or the physical hardware of the electronic device, and because the function and the attribute of the operation interface and/or the physical hardware simulated by the terminal device are the same as those of the operation interface and/or the physical hardware of the electronic device, the operation information obtained by the terminal device based on the operation interface and/or the physical hardware is the same as the operation information generated by the electronic device executing the same, and therefore the operation element information generated by the terminal device based on the operation information is the same as the operation element information generated by the electronic device executing the operation information, so that the terminal device can obtain the operation element information of the electronic device without the electronic device. In addition, the electronic equipment in the embodiment of the application does not need to share the own operation interface with the terminal equipment, so that the risk of revealing the privacy of the user of the electronic equipment is avoided

Referring to fig. 10, fig. 10 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application, where the method includes:

step S201, the first terminal device requests the first electronic device corresponding to the electronic device type identifier to establish a communication connection.

The first terminal device may establish a Communication connection with the first electronic device in a variety of manners such as a Near-Field Communication manner (e.g., WIFI (Wireless-Fidelity), bluetooth, NFC (Near Field Communication), etc.), a local area network, a distributed device network, etc.

Step S202, in a case that the first terminal device does not successfully establish a communication connection with the first electronic device, the first terminal device simulates an operation interface and/or physical hardware of the first electronic device.

Step S201 and step S202 are one possible implementation manner of step S101.

In the case of a plurality of first electronic devices, for example, electronic devices such as a PC, a TV, and a music box, the sequence of operations for the first electronic devices is as follows: PC, TV and audio box, the process of the first terminal device simulating the operation interface and/or physical hardware of the above three electronic devices is: the first terminal device simulates an operation interface and/or physical hardware of the PC and receives operation information input by a user on the simulated operation interface and/or physical hardware of the PC. After the first terminal device completes simulation of the operation interface and/or the physical hardware of the PC, the first terminal device simulates the operation interface and/or the physical hardware of the TV again, receives operation information input by a user on the operation interface and/or the physical hardware of the simulated TV, and completes simulation of the operation interface and/or the physical hardware of the TV. And the first terminal equipment simulates the operation interface and/or the physical hardware of the sound box and receives the operation information input by the user based on the simulated operation interface and/or the physical hardware of the sound box.

It should be noted that, the first terminal device may also simulate the operation interfaces and/or the physical hardware of the PC, the TV, and the sound box in sequence, and then receive the operation information input by the user on the simulated operation interfaces and/or the physical hardware of different electronic devices in sequence, which is not limited in this embodiment of the application.

As a possible implementation manner, the simulation of the operation interface and/or the physical hardware of the first electronic device by the first terminal device in the embodiment of the present application may be implemented by: the first terminal device may invoke a virtual device model of the first electronic device. Then, the first terminal device simulates the operation interface and/or the physical hardware of the first electronic device through the virtual device model.

The virtual device model of the first electronic device may be stored in the first terminal device, or may be stored in the cloud server, so that the first terminal device may obtain the virtual device model of the first electronic device from the cloud server, and of course, the virtual device model of the first terminal device may also be stored in the first electronic device, so that the first terminal device may also obtain the virtual device model of the first electronic device from the first electronic device.

For example, the virtual device model may include physical features, software features, operating component features of the first electronic device, and a 3D/2D device appearance/operation interface simulation model for presentation, operation in a virtual environment, and the like.

For example, the virtual device model may contain the information shown in table 1:

TABLE 1 information contained in the virtual appliance model

Referring to table 1, the virtual device model may contain device Identification (ID), device type, connection type, operation that may be recorded, hardware information, software information, 2D/3D model data, and the like. Wherein the device ID is an identification of the virtual device model. The device type is the device type of the first electronic device corresponding to the virtual device model. The connection type is a communication connection mode of the virtual device model, such as WIFI, Bluetooth, NFC, a local area network, a distributed device network and the like; the operation which can be recorded is used for limiting the operation which can be recorded in the operation of the virtual equipment model, and can be used for preventing privacy information from being leaked; the hardware information represents the physical characteristics of the first electronic equipment corresponding to the virtual equipment model; the software information characterizes software features of an application that may be run in the virtual device model. The 2D/3D model data is an appearance/operation interface simulation model and the like of the first electronic equipment corresponding to the virtual equipment model.

The first terminal device calls the virtual device model of the first electronic device, so that an operation interface and/or physical hardware of the first electronic device can be simulated in the first terminal device, namely, physical features, software features, operation component features, a 3D/2D device appearance/operation interface simulation model and the like of the first electronic device can be converted into the first terminal device to run, and operation element information generated by the first terminal device based on the operation information is the same as operation element information generated by the first electronic device executing the operation information, so that the first terminal device can obtain the operation element information of the first electronic device without the first electronic device.

Step S203 to step 205, refer to step S102 to step 104, and are not described herein again.

Step S203 may be implemented by: the first terminal equipment receives operation information which is input by a user and is based on the operation interface and/or the physical hardware.

Referring to fig. 11, fig. 11 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application, where the method may include:

step S301, see step S201, is not described herein again.

Step S302, under the condition that the first terminal device and the first electronic device successfully establish communication connection, the first terminal device sends a time synchronization request to the first electronic device.

The electronic device type identifier may be multiple, each electronic device type identifier corresponds to a first electronic device, each first electronic device corresponds to multiple sets of operation element information, and each set of operation element information corresponds to a generation time and an electronic device type identifier.

The time synchronization request is used for requesting the first electronic equipment to adjust the time of the first electronic equipment to be consistent with the time of the first terminal equipment. And each first electronic device responds to the time synchronization request and adjusts the time of the first electronic device to be consistent with the time of the first terminal device.

In an application scenario, the electronic device type identifier may be multiple, each electronic device type identifier corresponds to one first electronic device, the generation time of the operation element information of each first electronic device is determined based on the time of the corresponding first electronic device, so that there may be a difference in reference time of the operation element information of each first electronic device, the first electronic device is required to send the operation element information corresponding to each operation information to the first terminal device in real time, and the first terminal device performs packet grouping on each group of operation element information according to the time of receiving the operation element information and the electronic device type identifier corresponding to the operation element, so as to generate operation guidance information corresponding to each first electronic device, which may result in low data transmission efficiency.

Based on this problem, the time of each first electronic device is adjusted to be consistent in step S302, so that each first electronic device may not need to send the operation element information to the first terminal device in real time, thereby improving the data transmission efficiency.

Step S303, the first terminal device sends an operation element information acquisition request to the first electronic device.

The operation element information acquisition request is used for requesting the first electronic equipment to acquire operation element information generated by the first electronic equipment in response to the received operation information, and sending the operation element information to the first terminal equipment.

For example, in a case where the first terminal device successfully establishes a communication connection with first electronic devices such as a PC, a TV, a smart refrigerator, a sound box, and a router, the first terminal device transmits an operation element information acquisition request to each of the first electronic devices, respectively. The user can perform preset operation on the first electronic equipment, each first electronic equipment responds to the operation element information acquisition request, receives operation information input by the user and acquires operation element information generated by the first electronic equipment according to the operation information, and the first electronic equipment sends the operation element information to the first terminal equipment.

Illustratively, the operation element information may be recorded in the format shown in table 2:

table 2 operation element information recording format

Header

Node ID

Node Type

Operation ID

Time Stamp

Data

Tail

Data packet header

Node ID

Node type

Operation ID

Time stamp

Operating data

Data packet trailer

Referring to table 2, the specific format of the operation element information may include a data packet header, a node ID, a node type, an operation ID, a timestamp, operation data, and a data packet trailer. The node ID represents the electronic device corresponding to the operation element information, and may be set by the first terminal device in the process of performing communication connection with each first electronic device, and may correspond to the device ID in table 1; the node type represents the device type of the electronic device, and may be synchronized with the device type in table 1, for example, when the device type in table 1 is a PC, the node type in the operation element information generated by the PC device is the same as the device type in table 1; the operation ID is an operation identifier of the electronic equipment or the virtual equipment model, wherein a data structure corresponding to an operation corresponding to the operation ID can be extracted from the electronic equipment or the virtual equipment model according to the combination of the node type and the operation ID, and the combination can distinguish the possible differences of the electronic equipment with different operation interface sizes in the definition of the same operation; the time stamp is used for defining the time of executing the operation and can be used for recording the operation guide information.

For user privacy information related to interface operation elements, such as account number, password, fingerprint, face and other authentication information, the electronic device may perform matching identification through a pre-established high-risk operation blacklist, convert a process related to the privacy information related to the process into a preset operation ID, and for example, for information such as the account number and the password, convert a process of inputting the account number and the password into the preset operation ID to prevent leakage of the user privacy information.

Step S304, the first terminal device obtains the operation element information sent by the first electronic device.

Step S305, see step S104, will not be described herein again.

In a possible embodiment, the recording, by the first terminal device, the operation element information to generate the operation guidance information corresponds to performing time synchronization processing on the time of the electronic device, and may specifically include: the first terminal equipment determines the operation element information corresponding to each first electronic equipment according to the generation time and the electronic equipment type identification corresponding to each group of operation element information; the first terminal device packages the operation element information corresponding to each first electronic device, and generates operation guide information corresponding to each first electronic device.

After the time of the first electronic device is adjusted to be consistent with the time of the first terminal device, the first terminal device may determine the operation element information corresponding to each first electronic device according to the electronic device type identifier corresponding to the operation element information, and the first terminal device packages the operation element information corresponding to each first electronic device according to the sequence of the generation time of the operation element information to generate operation guidance information corresponding to each first electronic device, so that the data transmission efficiency can be improved.

Further, after the time of the first electronic device is adjusted to be consistent with the time of the first terminal device, the first electronic device may aggregate the operation element information in the first electronic device and then send the aggregated operation element information to the first terminal device, and the first terminal device packages the operation element information sent by each first electronic device to generate operation guidance information corresponding to each first electronic device. In an application scenario, when a user operates an electronic device on a first electronic device, supplementary description may be performed on operation information in a voice mode at the same time, so that operation guidance information sent by the first electronic device to the first terminal device may further include text information, where the text information is: and the first electronic equipment converts the acquired audio information used for carrying out operation guidance together with the operation information into text information in a time range corresponding to the acquired operation information.

In an application scenario, when a user operates on a first terminal device based on a simulated operation interface and/or physical hardware of a first electronic device, supplementary description may be simultaneously performed on operation information in a voice manner, for which, the method may further include:

in a time range corresponding to the operation information acquired by the first terminal equipment, the first terminal equipment acquires audio information for carrying out operation guidance together with the operation information;

the first terminal device records the audio information in the operation guidance information.

In order to avoid the risk of revealing the user voiceprint information, the first terminal device may convert the audio information into Text information by using a Natural Language Processing (NLP) technique and record the Text information in the operation guidance information, and the second terminal device may convert the Text information into audio information by using a Text To Speech (TTS) technique and play the audio information.

For example, the text information may include an electronic device type identifier and time information, where the electronic device type identifier is used to represent an electronic device corresponding to the text information, and the time information is used to represent time when the terminal device acquires the text information.

For example, after acquiring audio information of "please adjust the routing antenna to the XX angle" of the user, the first terminal device converts the audio information into text information by using the NLP technology, where the text information includes: an electronic device type identification (for XX model routing), time information (for the time when the audio information is acquired), and "please adjust the routing antenna to XX angle". When the second terminal device plays back the operation guidance information of the XX-type route, the second terminal device may convert "please adjust the routing antenna to the XX angle" into audio information by using a TTS technique, play the audio information at a time corresponding to the time information, and combine with other operation guidance information to implement operation guidance on the route.

In an application scenario, after the first terminal device obtains the operation guidance information of the electronic device, the first terminal device may display the operation guidance information, so that a user of the first terminal device can intuitively know the electronic device and the application program corresponding to the operation guidance information.

For example, each piece of operation element information may correspond to a generation time, an electronic device type identifier, and an application identifier, the first terminal device obtains the electronic device type identifier and the application identifier corresponding to the operation element information, establishes a link relationship between the operation element information and the corresponding electronic device type identifier and application identifier, and displays the electronic device type identifier and the application identifier in the first terminal device.

The first terminal device may divide the operation element information according to the electronic device type identifier and the application identifier, divide the operation element information corresponding to a certain electronic device type identifier according to the application identifier, establish a link relationship between the operation element information and the electronic device type identifier and the application identifier, and display the electronic device type identifier and the application identifier in a list or a chart or the like, so that a user can intuitively know the electronic device and the application corresponding to the operation guidance information. Meanwhile, after receiving the operation information of clicking the type identifier and/or the application identifier of the electronic device, the first terminal device may call corresponding operation element information based on the link relationship and play back the operation element information through the corresponding virtual device model, so that a user of the first terminal device may check whether the recorded operation guidance information is complete.

After receiving the input checking completion information, the first terminal device may package the operation element information and issue the operation element information to the cloud for security checking. The cloud end can perform security check on the operation element information related to the appointed APP in the operation element information and insert a security prompt into the operation element information for changing the sensitive setting of the system.

It should be noted that, after the first terminal device records the operation element information and generates the operation guidance information, the first terminal device may upload the operation guidance information to the cloud or send the operation guidance information to the second terminal device.

Referring to fig. 12, fig. 12 is a schematic flowchart of a method for recording operation guidance information of an electronic device according to an embodiment of the present application, where the method includes:

step S401, the second terminal device obtains operation guidance information for performing operation guidance on the second electronic device, where the operation guidance information includes an electronic device type identifier and operation element information.

The number of the electronic device type identifiers can be one or more.

For example, the second terminal device may be terminal device B shown in fig. 1 or fig. 2. The second electronic device and the first electronic device in this step may be electronic devices with types corresponding to each other, and please refer to relevant contents in fig. 1 or fig. 2 for details, which are not described herein again.

In one possible implementation, step S401 may be implemented by: and the second terminal equipment acquires the operation guide information from the cloud.

In another possible implementation manner, step S401 may be implemented by: the second terminal device acquires the operation guidance information from the first terminal device.

And S402, simulating an operation interface and/or physical hardware of the second electronic equipment corresponding to the electronic equipment type identifier by the second terminal equipment.

And step S403, the second terminal device generates prompt information for guiding the operation of the second electronic device based on the operation element information in the operation interface and/or the physical hardware response operation guidance information.

The second terminal device may extract and analyze the operation element information one by one according to the execution sequence of each operation element information in the operation guidance information, and respond to the analyzed operation element information based on the operation interface and/or the physical hardware simulated by the second terminal device.

Referring to (a) in fig. 13, the second terminal device displays at least the operation guidance APP300 as shown in (a) in fig. 13 through the display interface.

After the second terminal device detects that the user clicks the operation guidance APP300 shown in (a) in fig. 13, the second terminal device opens the operation guidance APP300, and displays an interface shown in (b) in fig. 13.

As shown in (b) in fig. 13, in response to the electronic device type identifier in the operation guidance information, the second terminal device displays controls of one or more electronic devices corresponding to the electronic device type identifier for the user to click. As shown in (c) of fig. 13, the user may click a control of one or more electronic devices in the display interface of the second terminal device, and then click a "confirmation" control, thereby enabling selection of the clicked one or more electronic devices. The second terminal device may simulate, in response to the user clicking the selected one or more electronic devices, the user clicking the operation interface and/or the physical hardware of the selected one or more electronic devices, as shown in (d) of fig. 13, and the second terminal device may simulate, in a second operation interface area 301 of the second terminal device, the user clicking the operation interface and/or the physical hardware of the selected one or more electronic devices to display operation guidance information of the electronic devices.

For example, taking the second electronic device as a personal computer as an example, as shown in (a) of fig. 14, the second terminal device displays an operation interface 301 of the simulated personal computer and prompt information of "please click" the control panel "displayed in the operation interface 301, after a certain time (for example, 5 seconds) has elapsed, the second terminal device displays the operation interface of the personal computer shown in (b) in fig. 14 and the prompt message of "please click" the program and function ", after a certain time (for example, 5 seconds) has elapsed, the second terminal device displays the operation interface of the personal computer shown in (c) in fig. 14 and the prompt message of "please click" Foxmail ", after a certain time (for example, 5 seconds) has elapsed, the second terminal device displays the operation interface of the personal computer shown in (d) in fig. 14 and the prompt message "please click" to unload ". The user can uninstall the Foxmail program in the personal computer according to the prompt message.

For example, taking the second electronic device as a router as an example, as shown in fig. 15, the second terminal device displays the simulated physical hardware of the router and a prompt message "please adjust the angle of the antenna in the XX direction", and the user can adjust the angle of the antenna of the router according to the prompt message.

The second terminal device may draw a highlighted frame at the edge of the key to be operated, and attach a related operation prompt, such as a text or an arrow, or a sliding animation identifier, where the operation guide may be a pre-recorded animation. And guiding the user to operate the physical key through the guide identifier displayed on the screen interface. If the application program corresponding to the operation element information is not detected in the electronic equipment, the electronic equipment can download the application program through a network and install the application program.

In the method for acquiring the operation guide information of the electronic equipment, after the second terminal equipment acquires the operation guide information for performing operation guide on the second electronic equipment, the operation interface and/or the physical hardware of the second electronic equipment corresponding to the type identifier of the electronic equipment is simulated, and the operation guide information is responded based on the operation interface and/or the physical hardware, the simulation of the second terminal equipment has the functions and the attributes of the operation interface and/or the physical hardware of the second electronic equipment, and the results of the second terminal equipment and the second electronic equipment executing the same operation information are the same, so that the operation guide effect of the second terminal equipment responding to the operation guide information through the simulated operation interface and/or the physical hardware is the same as the operation guide effect of the second electronic equipment responding to the operation guide information, therefore, the second terminal device can respond to the operation guide information without the second electronic device, and the user can be guided to operate the second electronic device.

As a possible embodiment, the step S402 may specifically include:

under the condition that the second electronic equipment is the electronic equipment without the operation guide information response unit, the second terminal equipment simulates an operation interface and/or physical hardware of the second electronic equipment; wherein, the operation guide information response unit comprises a display unit and/or an audio playing unit.

The second terminal device may determine whether the second electronic device corresponding to the electronic device type identifier has the operation guidance information response unit based on the electronic device type identifier. For convenience of description, the electronic device having the operation guidance information response unit is referred to as a third electronic device, and the electronic device not having the operation guidance information response unit is referred to as a fourth electronic device.

Illustratively, a second electronic device such as a PC, a TV, an intelligent refrigerator, a sound box, and a router is taken as an example, wherein a third electronic device such as the PC, the TV, the intelligent refrigerator, and the sound box has a display unit and/or an audio playing unit in an operation guidance information response unit, and a fourth electronic device such as the router does not have the operation guidance information response unit, so for operation guidance information corresponding to the fourth electronic device, the second terminal device simulates an operation interface and/or physical hardware of the fourth electronic device, and responds to the operation guidance information corresponding to the fourth electronic device based on the operation interface and/or the physical hardware.

For example, the second terminal device may call a virtual device model of the second electronic device, and simulate an operation interface and/or physical hardware of the second electronic device through the virtual device model. The virtual device model of the second electronic device may be stored in the terminal device, or may be stored in the cloud server, which is not limited in this embodiment of the present application. The virtual device model may include physical features, software features, operating component features of the electronic device, and 3D/2D device appearance/operation interface simulation models for presentation, operation in a virtual environment, etc., and the detailed description of the virtual device model may refer to table 1 and related contents.

In some embodiments, the operation guidance information may be operation guidance information corresponding to a plurality of second electronic devices, and therefore, the electronic device type identifier may be a plurality of electronic device type identifiers, each electronic device type identifier corresponds to one second electronic device, each second electronic device corresponds to one operation guidance information, each operation guidance information includes a plurality of sets of operation element information, and each set of operation element information corresponds to one generation time.

Correspondingly, the responding operation guidance information based on the operation interface and/or the physical hardware in step S403 may specifically include: the second terminal equipment analyzes the operation guide information corresponding to each second electronic equipment in sequence to obtain multiple groups of operation element information corresponding to each second electronic equipment; and the second terminal equipment responds to each group of operation element information sequentially through an operation interface and/or physical hardware of the second electronic equipment corresponding to each group of operation element information according to the sequence of the generation time corresponding to each group of operation element information.

For example, taking an electronic device such as a PC, a TV, and a sound box as an example, the sequence of the generation time of the operation element information of the electronic device is sequentially the operation element information corresponding to the PC, the operation element information corresponding to the TV, and the operation element information corresponding to the sound box, the second terminal device simulates an operation interface of the PC first, responds to the operation element information corresponding to the PC through the simulated operation interface of the PC to display corresponding prompt information, then simulates an operation interface of the TV, responds to the operation element information corresponding to the TV through the simulated operation interface of the TV to display corresponding prompt information, and finally simulates physical hardware of the sound box, and responds to the operation element information corresponding to the sound box through the simulated physical hardware of the sound box to display corresponding prompt information.

In some embodiments, the second terminal device responds to the operation guidance information based on the operation interface and/or the physical hardware, and specifically may be: the second terminal device may determine a position of the operation element information in the operation interface according to the operation interface layout in the operation element information, and display the prompt information drawn based on the operation element information at the position.

For example, when a display interface needing to be operated exists in operation element information corresponding to a certain operation guidance step in the operation guidance information, the second terminal device may extract key element information in the corresponding operation element information and determine a position of the key element information in the operation interface, and display prompt information drawn based on the operation element information in the simulated operation interface according to the position, for example, in a highlight frame or an arrow guide form; for the case that text information exists, the second terminal device may convert the text information into audio information for playing using TTS.

The second terminal device can detect the elements of the UI interface of the second electronic device operation interface simulated by the second terminal device through the system service, and can position the elements to the positions in the related application programs, virtual key operation, screen sliding, mobile phone posture adjustment, entity keys and the like according to the operation element information (such as information of software package names, relative positions of operation points, sequence of operation in a list and the like), so that visual prompts can be given to a user. In addition, for the operation guidance information corresponding to the third electronic device, the second terminal device may send the operation guidance information to the corresponding third electronic device, and the fourth electronic device responds to the operation guidance information, or the second terminal device may also simulate an operation interface and/or physical hardware of the third electronic device, and respond to the operation guidance information corresponding to the third electronic device based on the operation interface and/or physical hardware. Based on this, the above method may include: in the case where the second electronic device is an electronic device having an operation guidance information response unit, the second terminal device transmits operation guidance information to the second electronic device.

In some embodiments, the operation guidance information may include interface operation element information for an operation interface of the second electronic device, hardware operation element information for physical hardware of the second electronic device, and text information converted from audio information for performing operation guidance.

The interface operation element information is used for indicating a display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used for indicating an audio playing unit of the second electronic device to play the hardware operation element information, and the text information is used for indicating the audio playing unit of the second electronic device to convert the text information into audio information for playing. For example, for a scenario where a user operates the speaker to perform volume adjustment, the speaker may generate a hardware operation guidance element "adjust the volume adjustment key of the XX-type speaker", for which the speaker may play the audio information "please adjust the volume adjustment key" through the audio playing unit.

In some embodiments, for a second electronic device having an operation guidance information response unit, if the second terminal device cannot establish a communication connection with the second electronic device, the second terminal device simulates an operation interface and/or physical hardware of the electronic device, and responds to operation guidance information based on the operation interface and/or the physical hardware.

Further, the method may further include: the second terminal equipment responds to the received operation instruction and sends the operation instruction to the corresponding second electronic equipment so that the second electronic equipment can execute the operation instruction; the operation instruction may be an operation instruction based on the operation guidance information received by the second terminal device after the second terminal device responds to the operation guidance information based on the operation interface and/or the physical hardware.

It should be noted that, for GUI (graphical User interface, User graphical interaction) operation, the second terminal device may respond to the operation element information, the User may perform operation on the simulated operation interface and/or the physical hardware, the second terminal device converts the operation information of the User into an operation instruction and sends the operation instruction to the corresponding second electronic device, and the second electronic device executes the operation instruction; for physical hardware operations, such as key pressing, slider, knob, posture adjustment, antenna adjustment, etc., which must be completed by operating physical hardware, the second terminal device may generate a prompt message to remind the user of performing manual operation.

For example, the second terminal device may simulate an operation interface and/or physical hardware of the second electronic device in an AR/VR manner, and obtain an operation instruction and send the operation instruction to the corresponding second electronic device by combining technologies such as human body motion recognition, so that the second electronic device executes the operation instruction.

In an application scenario, after the second terminal device obtains the operation guidance information, the second terminal device may perform signature verification on the operation guidance information and check the integrity of the operation guidance information. After the signature verification and integrity check of the operation guide information pass, the second terminal device searches for second electronic equipment corresponding to the electronic equipment type identifier, and displays the searched second electronic equipment in a mode of an equipment list or an equipment icon and the like for a user to click.

If the number and the type of the second electronic device searched by the second terminal device are different from the number and the type of the electronic device type identifier in the operation guidance information, the second terminal device may generate a reminding message for reminding the user to check the working state of the corresponding second electronic device, or call a virtual device model of the second electronic device that is not searched, and play back the operation guidance information of the second electronic device through the virtual device model. In addition, the second terminal device may save the operation guidance information in a local or cloud space, so that the user may query the operation guidance information through relevant resolution software or a smart assistant or the like to execute the operation guidance information again.

An embodiment of the present invention further provides a method for recording operation guidance information of an electronic device, where the method may be applied to a first electronic device, and the method includes the following steps:

the method comprises the steps that first electronic equipment receives an operation element information acquisition request sent by first terminal equipment;

the first electronic equipment responds to the operation element information acquisition request, receives input operation information and acquires operation element information generated based on the operation information;

the first electronic equipment sends the operation element information to the first terminal equipment.

After receiving the operation element information acquisition request, the first electronic device may receive operation information generated by a user operating on an operation interface and/or physical hardware of the first electronic device. The operation element information may include interface operation element information for the operation interface and/or hardware operation element information for the physical hardware.

According to the method for recording the operation element information of the electronic equipment, after receiving an operation element information acquisition request sent by terminal equipment, first electronic equipment responds to the operation element information acquisition request, receives input operation information, acquires operation element information generated based on the operation information, and sends the operation element information to the first terminal equipment, so that the first terminal equipment records the operation element information to generate operation guide information corresponding to the electronic equipment, and compared with an operation interface image in the prior art, the operation element information does not relate to content which may reveal the user privacy of the electronic equipment, and therefore the risk of revealing the user privacy in the operation guide process of the electronic equipment can be reduced; the operation guide element information does not relate to the size of the operation interface, so that the method can be better suitable for electronic equipment with different operation interface sizes compared with an operation interface image in the traditional technology; and the space required by the operation element information is smaller than that of the operation interface image, so that the requirement of the electronic equipment on a network is lower in the process of transmitting the operation element information to the terminal equipment, and the time delay of data transmission can be reduced.

In some embodiments, the method may further comprise: the method comprises the steps that first electronic equipment receives a time synchronization request sent by first terminal equipment, and the time of the first electronic equipment is adjusted to be consistent with the time of the first terminal equipment based on the time synchronization request;

correspondingly, the sending, by the first electronic device, the operation element information to the first terminal device may specifically include: the first electronic equipment aggregates the operation element information generated by the first electronic equipment and then sends the operation element information to the first terminal equipment.

The embodiment of the present application further provides a method for responding to the operation guidance information of the electronic device, which can be applied to a second electronic device, and the method includes the following steps:

the second electronic equipment receives operation guide information sent by the second terminal equipment, wherein the operation guide information comprises a plurality of groups of operation element information, and each group of operation element information corresponds to a generation time;

and the second electronic equipment responds to the groups of operation element information in sequence based on the generation time of each group of operation element information.

According to the method for responding to the operation guide information of the electronic equipment, after the second electronic equipment receives the operation guide information sent by the second terminal equipment, the second electronic equipment responds to each group of operation element information in the operation guide information in sequence based on the generation time of each group of operation element information in the operation guide information, and as the operation element information is compared with an operation interface image in the traditional technology, the content which possibly reveals the user privacy of the electronic equipment is not involved, the risk of revealing the user privacy in the operation guide process of the electronic equipment can be reduced; and the operation guide element information does not relate to the size of the operation interface, so that the electronic equipment with different operation interface sizes can be better adapted to the operation interface image in the traditional technology.

In some embodiments, the operation element information includes interface operation element information for an operation interface of the second electronic device, and/or hardware operation element information for physical hardware of the second electronic device; the interface operation element information includes an operation interface layout and operation element information corresponding to a state change of the operation interface. The second electronic device displays the interface operation element information through the display unit, plays the hardware operation element information through the audio playing unit, and converts the text information into audio information through the audio playing unit for playing.

The second electronic device may specifically respond to the operation guidance information by: and the second electronic equipment determines the position of the operation element information in the operation interface according to the layout of the operation interface in the operation element information, and displays the prompt information drawn based on the operation element information at the position.

Fig. 16 shows a block diagram of an electronic device operation guidance information recording apparatus 500 according to an embodiment of the present application, which corresponds to the electronic device operation guidance information recording method applied to the first terminal device according to the foregoing embodiment.

Referring to fig. 16, the apparatus for recording operation guidance information of an electronic device in an embodiment of the present application may include a first simulation unit 501, a first receiving unit 502, a first obtaining unit 503, and a recording unit 504.

A first simulation unit 501, configured to simulate, in response to a received electronic device type identifier, an operation interface and/or physical hardware of a first electronic device corresponding to the electronic device type identifier;

a first receiving unit 502, configured to receive operation information based on an operation interface and/or physical hardware;

a first acquisition unit 503 configured to acquire operation element information generated based on the operation information;

the recording unit 504 is configured to record the operation element information and generate operation guidance information.

In the above electronic device operation guidance information recording apparatus, the first terminal device simulates the operation interface and/or physical hardware of the first electronic device corresponding to the electronic device type identifier in response to the received electronic device type identifier, receives the operation information based on the operation interface and/or physical hardware, and obtains the operation element information generated based on the operation information, and since the first terminal device simulates the operation interface and/or physical hardware of the first electronic device, the first terminal device has the functions and attributes of the operation interface and/or physical hardware of the first electronic device, and the results of the same operation information executed by both the first terminal device and the first electronic device are the same, the operation element information generated by the first terminal device based on the operation information is the same as the operation element information generated by the operation information executed by the first electronic device, therefore, the first terminal device can acquire the operation element information of the first electronic device without the first electronic device.

Corresponding to the method for acquiring electronic device operation guidance information applied to the second terminal device of the above embodiment, fig. 17 shows a block diagram of the apparatus 600 for acquiring electronic device operation guidance information provided in the embodiment of the present application, and for convenience of explanation, only the parts related to the embodiment of the present application are shown.

Referring to fig. 17, the apparatus for acquiring operation guidance information of an electronic device in an embodiment of the present application may include a second acquisition unit 601, a second simulation unit 602, and a first response unit 603.

The second obtaining unit 601 is configured to obtain operation guidance information for performing operation guidance on the second electronic device, where the operation guidance information includes an electronic device type identifier;

a second simulation unit 602, configured to simulate an operation interface and/or physical hardware of a second electronic device corresponding to the electronic device type identifier;

a first response unit 603, configured to respond to the operation guidance information based on the operation interface and/or the physical hardware.

In the apparatus for acquiring operation guidance information of an electronic device, after acquiring operation guidance information for performing operation guidance on the second electronic device, the second terminal device simulates an operation interface and/or physical hardware of the second electronic device corresponding to the type identifier of the electronic device, and responds to the operation guidance information based on the operation interface and/or physical hardware, and the second terminal device simulates the function and attribute of the operation interface and/or physical hardware of the second electronic device, and the results of executing the same operation information by both the second terminal device and the second electronic device are the same, so that the operation guidance effect of the second terminal device responding to the operation guidance information through the simulated operation interface and/or physical hardware is the same as the operation guidance effect of the second electronic device responding to the operation guidance information, therefore, the second terminal device can respond to the operation guide information without the second electronic device, and the user can be guided to operate the electronic device.

Corresponding to the method for recording the electronic device operation guidance information applied to the first electronic device in the above embodiments, the embodiments of the present application further provide an apparatus for recording the electronic device operation guidance information, and for convenience of description, only the parts related to the embodiments of the present application are shown.

The electronic device operation guidance information recording device in the embodiment of the application may include a second receiving unit, a second responding unit, and a transmitting unit. The second receiving unit is used for receiving an operation element information acquisition request sent by the first terminal equipment; the second response unit is used for responding to the operation element information acquisition request, receiving the input operation information and acquiring the operation element information generated based on the operation information; the sending unit is used for sending the operation element information to the first terminal equipment.

According to the electronic equipment operation guide information recording device, after receiving an operation element information acquisition request sent by first terminal equipment, first electronic equipment responds to the operation element information acquisition request, receives input operation information, acquires operation element information generated based on the operation information, and sends the operation element information to the first terminal equipment, so that the first terminal equipment records the operation element information to generate operation guide information corresponding to the first electronic equipment, and because the operation element information does not relate to content which may reveal user privacy of the electronic equipment compared with an operation interface image in the traditional technology, the risk of revealing the user privacy in the operation guide process of the electronic equipment can be reduced; the operation guide element information does not relate to the size of the operation interface, so that the method can be better suitable for electronic equipment with different operation interface sizes compared with an operation interface image in the traditional technology; and the space required by the operation element information is smaller than that of the operation interface image, so that the requirement of the electronic equipment on a network is lower in the process of transmitting the operation element information to the terminal equipment, and the time delay of data transmission can be reduced.

Corresponding to the method for responding to the electronic device operation guidance information applied to the second electronic device in the above embodiments, the present application embodiment also provides a device for responding to the electronic device operation guidance information, and for convenience of explanation, only the parts related to the present application embodiment are shown.

The device for responding to the electronic equipment operation guide information in the embodiment of the application can comprise a third receiving unit and a third responding unit. The third receiving unit is used for receiving operation guide information sent by the second terminal device, wherein the operation guide information comprises a plurality of groups of operation element information, and each group of operation element information corresponds to a generation time; and the third response unit is used for sequentially responding to each group of operation element information based on the generation time of each group of operation element information.

According to the electronic equipment operation guide information recording device, after the second electronic equipment receives the operation guide information sent by the second terminal equipment, the second electronic equipment responds to each group of operation element information in the operation guide information in sequence based on the generation time of each group of operation element information in the operation guide information, and as the operation element information is compared with an operation interface image in the traditional technology, the content that the user privacy of the electronic equipment is possibly leaked is not related, the risk that the user privacy is leaked in the operation guide process of the electronic equipment can be reduced; and the operation guide element information does not relate to the size of the operation interface, so that the electronic equipment with different operation interface sizes can be better adapted to the operation interface image in the traditional technology.

An embodiment of the present application further provides a terminal device, and referring to fig. 18, the terminal device 700 may include: at least one processor 710, a memory 720, and a computer program stored in the memory 720 and operable on the at least one processor 710, the processor 710, when executing the computer program, implements the steps of any of the above-described method embodiments, such as steps S101 to S104 in the embodiment shown in fig. 6, or steps S401 to S403 in the embodiment shown in fig. 12. Alternatively, the processor 710, when executing the computer program, implements the functions of each module/unit in each device embodiment described above, such as the functions of the modules 501 to 504 shown in fig. 16 or the functions of the modules 601 to 603 shown in fig. 17.

Illustratively, the computer program may be divided into one or more modules/units, which are stored in the memory 720 and executed by the processor 710 to accomplish the present application. One or more modules/units may be a series of computer program segments capable of performing certain functions, which are used to describe the execution of the computer program in the terminal device 700.

Those skilled in the art will appreciate that fig. 18 is merely an example of a terminal device and is not limiting of terminal devices and may include more or fewer components than shown, or some components in combination, or different components such as input output devices, network access devices, buses, etc.

The Processor 710 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 720 may be an internal storage unit of the terminal device, or may be an external storage device of the terminal device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like. The memory 720 is used for storing computer programs and other programs and data required by the terminal device. The memory 720 may also be used to temporarily store data that has been output or is to be output.

The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended ISA (EISA) bus, or the like. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the buses in the figures of the present application are not limited to only one bus or one type of bus.

Embodiments of the present application also provide a computer-readable storage medium having stored therein instructions, which when executed on a computer or processor, cause the computer or processor to perform one or more steps of any one of the methods described above.

Embodiments of the present application also provide a computer program product comprising instructions for causing a computer or processor to perform one or more steps of any one of the methods described above when the computer program product is run on the computer or processor.

Embodiments of the present application further provide a chip system, which may include a memory and a processor, where the processor executes a computer program stored in the memory to implement one or more steps of any one of the methods described above. The chip system can be a single chip or a chip module consisting of a plurality of chips.

Embodiments of the present application further provide a chip system, which may include a processor coupled with a memory, and the processor executes a computer program stored in the memory to implement one or more steps of any of the above methods. The chip system can be a single chip or a chip module consisting of a plurality of chips.

In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the present application are all or partially generated when the computer program instructions are loaded and executed on a computer. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on or transmitted over a computer-readable storage medium. The computer instructions may be transmitted from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optics, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a Solid State Disk (SSD)), among others.

One of ordinary skill in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by hardware related to instructions of a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the above method embodiments. And the aforementioned storage medium includes: various media capable of storing program codes, such as ROM or RAM, magnetic or optical disks, etc.

The above description is only a specific implementation of the embodiments of the present application, but the scope of the embodiments of the present application is not limited thereto, and any changes or substitutions within the technical scope disclosed in the embodiments of the present application should be covered by the scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for recording operation guide information of electronic equipment is characterized by comprising the following steps:

the method comprises the steps that a first terminal device responds to a received electronic device type identifier and simulates an operation interface and/or physical hardware of a first electronic device corresponding to the electronic device type identifier;

the first terminal equipment receives operation information based on the operation interface and/or the physical hardware;

the first terminal equipment acquires operation element information generated according to the operation information;

and the first terminal equipment records the operation element information to generate operation guide information, wherein the operation guide information comprises an electronic equipment type identifier.

2. The method according to claim 1, wherein the simulating, by the first terminal device, the operation interface and/or physical hardware of the first electronic device corresponding to the electronic device type identifier in response to the received electronic device type identifier comprises:

the first terminal equipment requests to establish communication connection with the first electronic equipment;

and under the condition that the first terminal equipment does not successfully establish communication connection with the first electronic equipment, simulating the operation interface and/or the physical hardware of the first electronic equipment by the first terminal equipment.

3. The method of claim 2, wherein the simulating, by the first terminal device, the operation interface and/or the physical hardware of the first electronic device comprises:

the first terminal device calls a virtual device model of the first electronic device, and the operation interface and/or the physical hardware of the first electronic device are/is simulated through the virtual device model.

4. The method of claim 2, wherein the method comprises:

the method comprises the steps that under the condition that the first terminal device and the first electronic device successfully establish communication connection, the first terminal device obtains operation element information of the first electronic device from the first electronic device.

5. The method according to claim 4, wherein the first terminal device obtains the operation element information of the first electronic device from the first electronic device, and comprises:

the first terminal device sends an operation element information acquisition request to the first electronic device, where the operation element information acquisition request is used to request the first electronic device to acquire operation element information generated by the first electronic device in response to the received operation information, and sends the operation element information to the first terminal device.

6. The method according to claim 4, wherein the electronic device type identifier is plural, each electronic device type identifier corresponds to a first electronic device, each first electronic device corresponds to plural sets of operation element information, and each set of operation element information corresponds to a generation time and an electronic device type identifier;

the method further comprises the following steps:

the first terminal device sends a time synchronization request to each first electronic device corresponding to the electronic device type identifiers, wherein the time synchronization request is used for requesting the first electronic device to adjust the time of the first electronic device to be consistent with the time of the first terminal device;

the first terminal device records the operation element information to generate operation guide information, and the operation guide information includes:

the first terminal equipment determines operation element information corresponding to each first electronic equipment in each first electronic equipment according to the electronic equipment type identification corresponding to each group of operation element information;

and the first terminal equipment packs the operation element information corresponding to each first electronic equipment according to the sequence of the generation time of the operation element information to generate operation guide information corresponding to each first electronic equipment.

7. The method according to claim 6, wherein the operation guidance information further includes text information, and the text information is: and the first electronic equipment converts the acquired audio information which is used for carrying out operation guidance together with the operation information into text information in a time range corresponding to the acquired operation information.

8. The method according to claim 1 or 4, wherein the operation element information comprises interface operation element information for an operation interface of the electronic device and/or hardware operation element information for physical hardware of the electronic device;

the interface operation element information comprises operation interface layout and operation element information corresponding to the state change of the operation interface.

9. The method of claim 1, further comprising:

in a time range corresponding to the operation information acquired by the first terminal device, the first terminal device acquires audio information for performing operation guidance together with the operation information;

and the first terminal equipment records the audio information in the operation guide information.

10. The method of claim 8, wherein the first terminal device records the audio information in the operation guidance information, and comprises:

and the first terminal equipment converts the audio information into text information and records the text information in the operation guide information.

11. A method for obtaining operation guide information of an electronic device is characterized by comprising the following steps:

the second terminal equipment acquires operation guide information for carrying out operation guide on second electronic equipment, wherein the operation guide information comprises electronic equipment type identification and operation element information;

the second terminal equipment simulates an operation interface and/or physical hardware of the second electronic equipment corresponding to the electronic equipment type identifier;

and the second terminal equipment responds to the operation element information based on the operation interface and/or the physical hardware to generate prompt information for carrying out operation guidance on the second electronic equipment.

12. The method according to claim 11, wherein the simulating, by the second terminal device, the operation interface and/or the physical hardware of the second electronic device corresponding to the electronic device type identifier includes:

under the condition that the second electronic equipment is electronic equipment without an operation guide information response unit, the second terminal equipment simulates the operation interface and/or the physical hardware of the second electronic equipment;

wherein the operation guidance information response unit comprises a display unit and/or an audio playing unit.

13. The method according to claim 12, wherein the second terminal device simulates the operation interface and/or the physical hardware of the second electronic device, and comprises:

and the second terminal equipment calls a virtual equipment model of the second electronic equipment, and simulates the operation interface and/or the physical hardware of the second electronic equipment through the virtual equipment model.

14. The method according to claim 12, wherein the electronic device type identifier is plural, each electronic device type identifier corresponds to a second electronic device, each second electronic device corresponds to an operation guidance information, each operation guidance information includes plural sets of operation element information, and each set of operation element information corresponds to a generation time;

the responding to the operation guiding information based on the operation interface and/or the physical hardware comprises the following steps:

the second terminal equipment analyzes the operation guide information corresponding to each second electronic equipment in sequence to obtain multiple groups of operation element information corresponding to each second electronic equipment;

and the terminal equipment responds to each group of operation element information sequentially through an operation interface and/or physical hardware of the second electronic equipment corresponding to each group of operation element information according to the sequence of the generation time corresponding to each group of operation element information.

15. The method of claim 12, wherein the method comprises:

and if the second electronic equipment is the electronic equipment with the operation guide information response unit, the second terminal equipment sends the operation guide information to the second electronic equipment.

16. The method according to claim 15, wherein the operation guidance information includes interface operation element information for a user interface of the second electronic device, hardware operation element information for physical hardware of the second electronic device, and text information converted from audio information for operation guidance;

the interface operation element information is used for indicating a display unit of the second electronic device to display the interface operation element information, the hardware operation element information is used for indicating an audio playing unit of the second electronic device to play the hardware operation element information, and the text information is used for indicating the audio playing unit of the second electronic device to convert the text information into audio information for playing.

17. A terminal device, comprising: one or more processors, memory, and a display screen;

the memory, the display screen, and the one or more processors, the memory to store computer program code, the computer program code comprising computer instructions;

the computer instructions, when executed by the one or more processors, cause the terminal device to perform the method of any of claims 1-10, or the method of any of claims 11-16.

18. A chip system, characterized in that the chip system comprises a processor coupled with a memory, the processor executing a computer program stored in the memory to implement the method of any of claims 1 to 10, and/or the method of any of claims 11 to 16.