TWI740361B - Artificial intelligence operation assistive system and method thereof - Google Patents

Abstract

An artificial intelligence operation assistive system comprises a portable device and an artificial intelligence operation assistive platform. The portable device comprises a camera module, a wireless communication module, a processor, and a display module. The camera module is configured to obtain an environment image with a target object. The wireless communication module is configured to send the environment image and to receive an operation signal. The processor is configured to control the wireless communication module to send the environment image and generate an operation layer according to the operation signal. The display module is configured to display the operation layer. The artificial intelligence operation assistive platform is configured to receive the environment image and perform a recognition procedure to identify the target object, and generate and send the operation signal according to the target object and a object database.

Description

Intelligent operation auxiliary system and intelligent operation auxiliary method

The present invention relates to image recognition, in particular to a smart operation assistance system and method based on Mixed Reality (MR) technology and high-speed communication network.

In the petrochemical or chemical industry plants, there are complex pipelines, a wide variety of equipment and various instruments. In order to maintain the safety of various assets and equipment in the plant, it is necessary to deploy enough personnel to conduct inspections to facilitate daily routine inspections. Monitoring of equipment index data, and at the same time eliminate emergency events.

However, for new patrol personnel, because they cannot be immediately familiar with the operation methods and troubleshooting mechanisms of all equipment, operating errors often occur. Even if the inspectors use the walkie-talkie to talk to the central control center to get instructions on the operation, it is still possible that the operator may make mistakes in the operation due to the lack of clear meaning of the two parties, and cause the fault to be unable to be eliminated or the equipment to fail. This may result in unpredictable losses for the factory.

Accordingly, the present invention provides a kind of wisdom and intelligence operations support system operations helper method implemented in dedicated spectrum under the fifth generation of mobile communications technology ^{(5 th Generation mobile networks, 5G} ) corporate network deployment environment made of high-speed communication The equipment operation assistance guidance in the petrochemical industry plant area can effectively enable new inspectors to operate correctly when facing the complex equipment and instruments in the petrochemical industry plant, reducing the possibility of human error.

A smart operation assistance system described according to an embodiment of the present invention includes a portable device and an artificial intelligence operation assistance platform. The portable device includes a camera module, a wireless communication module, a processor, and a display module. The camera module is used to obtain environmental images. There is a target object in the environment image. The wireless communication module is used to send environmental images and receive operation signals. The processor is electrically connected to the wireless communication module and the camera module. The processor controls the wireless communication module to send environmental images, and generates an operation layer according to the operation signal. The display module is used for presenting the operation layer. The artificial intelligence operation auxiliary platform communicates with the wireless communication module. The artificial intelligence operation auxiliary platform is used to receive environmental images and execute a recognition process to identify the target object. The artificial intelligence operation auxiliary platform is used to generate and generate data based on the target object and the object database. Send operation signal.

A smart operation assistance method described according to an embodiment of the present invention includes: a camera module of a portable device obtains an environmental image with a target object in the environmental image; and a processor of the portable device controls the wireless communication of the portable device The module sends environmental images; the artificial intelligence operation auxiliary platform receives the environmental images and executes the identification process to identify the target object; the artificial intelligence operation auxiliary platform generates and sends operation signals according to the target object and the object database; the wireless communication module of the portable device Receive the operation signal; the processor of the portable device generates an operation layer according to the operation signal; and a display module of the portable device presents the operation layer.

The above description of the disclosure and the following description of the embodiments are used to demonstrate and explain the spirit and principle of the present invention, and to provide a further explanation of the scope of the patent application of the present invention.

The detailed features and advantages of the present invention will be described in detail in the following embodiments. The content is sufficient to enable anyone familiar with the relevant art to understand the technical content of the present invention and implement it accordingly, and according to the content disclosed in this specification, the scope of patent application and the drawings. Anyone who is familiar with relevant skills can easily understand the purpose and advantages of the present invention. The following examples further illustrate the viewpoints of the present invention in detail, but do not limit the scope of the present invention by any viewpoint.

Please refer to FIG. 1, which shows a block diagram of a smart operation assistance system 100 according to an embodiment of the present invention. The smart operation assistance system 100 is suitable for petrochemical plants where a high-speed mobile communication network N has been deployed. In one embodiment, the high speed mobile communication network N by the radio access point AP and a plurality of base stations are composed of the wireless access point AP through the fifth-generation mobile communication system technologies ^{(5 th Generation mobile networks, 5G} ) dedicated spectrum of Enterprise private networks are connected to each other. As shown in FIG. 1, the intelligent operation assistance system 100 includes: a portable device 3 and an artificial intelligence operation assistance platform 5. The portable device 3 is connected to the artificial intelligence operation auxiliary platform 5 through one of the aforementioned wireless access points AP.

As shown in FIG. 1, the portable device 3 used for transmitting the environmental images visually seen by the inspectors includes a camera module 32, a wireless communication module 34, a processor 36 and a display module 38. The processor 36 is electrically connected to the camera module 32, the wireless communication module 34 and the display module 38. The artificial intelligence operation auxiliary platform 5 is communicatively connected to the wireless communication module 34 to receive environmental images. In one embodiment, the artificial intelligence operation assistance platform 5 is a server or a cluster of servers, and is communicatively connected to a high-speed mobile communication network N implemented by 5G. In this way, the artificial intelligence operation assistance platform 5 can receive the environmental images returned by each portable device 3 with a relatively small delay.

Please refer to FIG. 2, which shows a schematic diagram of the appearance of the portable device 3 described in an embodiment of the present invention. In one embodiment, the portable device 3 is a smart helmet worn by inspectors. Since the camera module 32 is arranged close to the eye position of the inspector wearing the portable device 3 of the smart helmet type, the inspector can easily collect the factory buildings he sees through the camera module 32 Images of various equipment inside. In other embodiments, the appearance of the portable device 3 can also be smart glasses, a smart phone with a camera module 32, a tablet computer with a camera module 32, or a personal digital assistant with a camera module 32. (Personal Digital Assistant, PDA) etc. The present invention does not limit the appearance of the portable device 3 or the type of hardware.

In the embodiment shown in FIG. 2, the display module 38 is disposed at the position of the lens. The display module 38 is a small transparent screen. Therefore, the inspectors can pass through the display module 38 to see the equipment inside the factory. At the same time, the inspector can also see the display layer presented on the small screen in the form of the lens. Therefore, in this embodiment, the display module 38 can provide mixed reality (Miexed Reality, MR) or augmented reality. The superimposed images presented in the form of Augemented Reality (AR) are available for inspection by inspectors. In another embodiment, the display module 38 is, for example, a general screen, such as the display panel of a tablet computer, and the user can see the actual image captured by the camera module 32 on the display module 38, or see The three-dimensional object image sent by the artificial intelligence operation auxiliary platform 5. The present invention does not limit the hardware type of the display module 38.

Please refer to FIG. 3, which shows a flowchart of a smart operation assistance method described in an embodiment of the present invention. The aforementioned smart operation assistance method is applicable to the aforementioned smart operation assistance system 100. The following describes the composition and implementation of various components in the smart operation assistance system 100 in conjunction with the flow of the smart operation assistance method shown in FIG. 3.

Please refer to step S1, the camera module 32 of the portable device 3 obtains and sends an environment image. There is a target object in the environment image. The target object generally refers to equipment or instruments that can be operated by inspectors. For example, an inspector wearing a portable device 3 as shown in FIG. 2 reaches a designated coordinate point and starts to inspect or operate petrochemical pipelines, equipment or instruments around the coordinate point. First, the camera module 32 of the portable device 3 obtains the environment image with the target object, and then, through the aforementioned high-speed mobile communication network N realized by the 5G enterprise private network, the processor 36 of the portable device 3 controls the portable The wireless communication module 34 of the mobile device 3 sends the environment image with the target object to the artificial intelligence operation assistance platform 5.

Please refer to step S2, the artificial intelligence operation auxiliary platform 5 recognizes the target object in the environmental image. In one embodiment, the artificial intelligence operation assistance platform 5 can execute an image recognition process and load an object database to identify the target object in the environmental image. The object database includes the image information of all equipment in the petrochemical industry plant, for example, in the form of a computer-aided design (CAD) file. Therefore, no matter what perspective the inspector views the target object, the artificial intelligence operation assistance platform 5 can recognize the type of the target object in the environmental image.

In another embodiment of step S2, when the target object has a readable value (for example, a meter on a pipeline), the identification procedure executed by the artificial intelligence operation auxiliary platform 5 further includes confirming the readable value. For example, the inspector confirms the readable value by himself, and then inputs the readable value through the input element of the portable device 3 and sends it back to the artificial intelligence operation auxiliary platform 5. Alternatively, the inspector can also read this value through other instruments and then send it back to the artificial intelligence operation auxiliary platform 5. Or, the instrument belongs to the type of Intelligent Internet of Things (AIoT), which automatically returns the readable value to the artificial intelligence operation auxiliary platform 5 through the high-speed communication network N every fixed cycle. Alternatively, the artificial intelligence operation assistance platform 5 directly recognizes the readable value from the environmental image captured by the camera module 32. In summary, the present invention does not limit the manner in which the smart operation assistance platform 5 can confirm the readable value.

In still another embodiment of step S2, before the artificial intelligence operation assistance platform 5 executes the identification process to identify the target object, it further includes identifying the identifiable objects around the target object. For example, the artificial intelligence operation assistance platform 5 loads the 3D model database, and then executes the identification process based on the environment image and the 3D model database to find all the identifiable objects in the environment image. According to the distribution of these identifiable objects, The artificial intelligence operation auxiliary platform 5 compares the object distribution information stored in the 3D model database to confirm the coordinate point of the current position of the inspector. If the artificial intelligence operation assistance platform 5 determines that there is the target object around the coordinate point where the inspector is located, the artificial intelligence operation assistance platform 5 can send a notification signal, thereby instructing the inspector's line of sight toward the location of the target object. Regarding the aforementioned 3D model database, it includes one or more Computer Aided Design (CAD) files and multiple coordinate points. For example, the CAD file of the petrochemical plant during the design can be used to create this 3D model database. In other embodiments, laser scanning or detailed construction drawings of a petrochemical plant can be used to reversely build a three-dimensional model database of pipelines and other equipment. For example, use a three-dimensional scanning instrument to obtain the spatial information of the pipeline or equipment surface, and use point cloud recognition software to convert it into a solid model with physical meaning. For another example, find out the coordinates of the start and end points or reference points of the pipeline according to the detailed construction drawings, and then semi-automatically build a 3D model database by the CAD software system. As mentioned above, after the inspectors are confirmed and their line of sight is facing the target object, the artificial intelligence operation auxiliary platform 5 performs the identification process to identify the target object according to the environment image and location information. Because the pipelines, equipment, and meters in the petrochemical plant are basically fixed, all equipment belonging to the location information can be screened out in advance through the above mechanism, so the time for the identification process to identify the target object can be reduced.

Please refer to step S3, the artificial intelligence operation assistance platform 5 generates and sends operation signals according to the target object and the object database. In detail, after confirming the target object in step S2, the artificial intelligence operation assistance platform 5 further outputs the operation signal applicable to the target object according to the target object and the object database. In one embodiment, the object database not only stores the images of all target objects, but also stores the target object state judgment model applicable to each target object. In practice, the artificial intelligence operation assistance platform 5 pre-collects a number of state parameters associated with the target object, such as the factory specifications of the target object, the numerical items that can be adjusted, and the display of the target object at different points in time. The status (such as temperature, pressure or meter readings, etc.). The artificial intelligence operation assistance platform 5 uses the collected state parameters associated with the target object as the output layer, and substitutes it into a neural network algorithm to generate a target object state judgment model, the output of which is the aforementioned operation parameter. The neural network-like algorithms are, for example, MultiLayer Perceptron (MLP), Long Short-Term Memory (LSTM), and Convolutional Neural Network (CNN).

In another embodiment, the artificial intelligence operation assistance platform 5 sets operation parameters according to readable values. For example, when the temperature of the device exceeds a first threshold, the operating valve associated with temperature control on the device is rotated to a specified angle to adjust the temperature, or when the pressure gauge of the device is lower than a first threshold. At the second threshold, the boost switch on the device is activated. It should be noted that the artificial intelligence operation assistance platform 5 of the present invention can dynamically adjust the configuration values of the operation parameters according to the current state of the equipment, so that the operation signal is suitable for the current situation of the inspector.

Please refer to step S4 and step S5. The portable device 3 receives and generates an operation layer according to the operation signal, and the display module 38 of the portable device 3 presents the operation layer. In detail, the processor 36 of the portable device 3 generates an operation layer according to the operation signal and the operation parameters therein. The operation layer is, for example, text messages or graphic messages. In other embodiments, the processor 36 further generates an operating voice according to the operating signal, and plays the voice through the speaker on the portable device 3.

In one embodiment, the display module 38 is a non-transparent display panel, which is used to present a superimposed image of the operation layer and the environment image. The environment image described here may be a real-life image captured by the camera module 32 , The operation layer is overlaid on the real image to form a superimposed image, so the effect of augmented reality (AR) can be achieved. The environment can also be a three-dimensional object image of the target object sent by the artificial intelligence operation auxiliary platform 5. The operation layer is overlaid on the virtual image to form a superimposed image, so that virtual reality (VR) or mixed reality (MR) can be achieved. )Effect. In another embodiment, the display module 38 is a display panel with transparency, which is used for presenting a screen of the operation diagram stacked on the physical target object. In the above embodiment, the portable device 3 further includes an inertial measurement unit (IMU), which is used to sense the line-of-sight angle of the inspector to the target object, and feed this angle back to the processor 36 . The processor 36 is further used to adjust the presentation type of the operating layer according to the viewing angle, so that the virtual image presented by the display module 38 and the target object can be presented with each other in a better form.

Please refer to FIG. 4, which illustrates a schematic diagram of the inspector viewing the operation layer L of an embodiment of the present invention through the display module 38. As shown in Figure 4, the inspector can see the physical target object (including the pipeline 71, the meter 73, the ball valve 75 and the handle 77) through the display module 38 of the transparent lens, and can see the display module 38 to mix at the same time. Mixed Reality is displayed on the operation layer L above the physical devices 71~77 (the display module 38 presents this operation layer in a semi-transparent form, for example), and then follow the instructions of the operation signal to move the handle 77 from the starting position of the operation A rotates to the operation end position B.

In summary, the intelligent operation assistance system proposed by the present invention can improve the accuracy of equipment operation by inspectors and improve industrial safety. And according to the current status of the equipment, the operation instructions are dynamically adjusted, the inspection information is collected and feedback, and the assets and equipment in the petrochemical plant are monitored and managed.

Although the present invention is disclosed in the foregoing embodiments, it is not intended to limit the present invention. All changes and modifications made without departing from the spirit and scope of the present invention fall within the scope of the patent protection of the present invention. For the scope of protection defined by the present invention, please refer to the attached scope of patent application.

100...Smart operation assistance system 3...portable device 32...Camera module 34...Wireless communication module 36...Processor 38...Display module 5...Auxiliary platform for artificial intelligence operation N...High-speed mobile communication network AP...Wireless access point S1~S5...Step A …operation starting position B...operation end position 71...Pipeline 73...Instrument 75...ball raft 77...handle L...Operation layer

FIG. 1 is a block diagram of a smart operation assistance system according to an embodiment of the present invention. FIG. 2 is a schematic diagram showing the appearance of a portable device according to an embodiment of the present invention. FIG. 3 is a flowchart of a method for assisting smart operation according to an embodiment of the present invention. FIG. 4 is a schematic diagram of operating layers according to an embodiment of the present invention.

100...Smart operation assistance system 3...portable device 32...Camera module 34...Wireless communication module 36...Processor 38...Display module 5...Auxiliary platform for artificial intelligence operation N...High-speed mobile communication network AP...Wireless access point

Claims (6)

A smart operation assistance system includes: a portable device, including: a camera module for obtaining an environment image, the environment image has a target object; a wireless communication module for sending the environment image, And receives an operation signal; an inertial measurement unit for obtaining a shooting angle of the camera module; a processor electrically connected to the wireless communication module and the camera module, and the processor controls the wireless communication module Send the environment image, and generate an operation layer according to the operation signal and the shooting angle; and a display module electrically connected to the processor, the display module for presenting the operation layer; and an artificial intelligence operation auxiliary platform , The wireless communication module is communicatively connected, the artificial intelligence operation auxiliary platform is used to receive the environment image and execute a recognition process to recognize the target object and multiple identifiable objects around the target object, the artificial intelligence operation auxiliary platform further It is used to generate and send the operation signal according to the target object and an object database; the artificial intelligence operation auxiliary platform further confirms the coordinate position of the portable device according to the identifiable objects and an object distribution information, and according to the coordinates The location judges whether there is the target object; wherein the target object has a readable value, the operation signal further includes an operation parameter, and the identification program is used to identify the readable value, and the artificial intelligence operation auxiliary platform is more useful To set the operating parameter according to the readable value; The artificial intelligence operation assistance platform is further used to collect a plurality of state parameters associated with the target object, and generate a target object state judgment model based on the state parameters and a type of neural network algorithm, and the output of the target object state judgment model Is the operating parameter. The operation of the auxiliary system according to the wisdom of a request for a high-speed network environment, the network environment includes a plurality of high-speed wireless access point, wireless access point of the plurality of fifth-generation mobile communication technology applications (5 ^th Generation Mobile networks (5G) dedicated spectrum enterprise private networks are connected to each other, and the wireless communication module is connected to the artificial intelligence operation auxiliary platform through one of the wireless access points. The intelligent operation assistance system according to claim 1, wherein the display module is further used for presenting a superimposed image of the operation layer and the environment image. A method for assisting smart operation, including: obtaining an environment image with a camera module of a portable device, the environment image having a target object; obtaining an image of the camera module with an inertial measurement unit of the portable device A shooting angle; a processor of the portable device controls a wireless communication module of the portable device to send the environment image; an artificial intelligence operation auxiliary platform receives the environment image and executes an identification process to identify the Target object; use the artificial intelligence operation auxiliary platform to generate and send an operation signal based on the target object and an object database; use the wireless communication module of the portable device to receive the operation signal; The processor of the portable device generates an operation layer according to the operation signal and the shooting angle; and a display module of the portable device presents the operation layer; wherein the target object has a readable value , The operation signal further includes an operation parameter, and the identification program is further used to identify the readable value, the artificial intelligence operation auxiliary platform is further used to set the operation parameter according to the readable value; the artificial intelligence operation auxiliary platform It is further used to collect a plurality of state parameters associated with the target object, and generate a target object state judgment model based on the state parameters and a type of neural network algorithm, and the output of the target object state judgment model is the operating parameter; Before receiving the environment image by the artificial intelligence operation auxiliary platform and executing the recognition procedure to identify the target object, it further includes: performing the recognition procedure on the artificial intelligence operation auxiliary platform to recognize a plurality of identifiable objects around the target object; The artificial intelligence operation assistance platform confirms the coordinate position of the portable device according to the identifiable objects and an object distribution information, and judges whether the target object is present according to the coordinate position. The intelligence operation assisting method according to a request item for a high-speed network environment, the network environment includes a plurality of high-speed wireless access point, wireless access point of the plurality of fifth-generation mobile communication technology applications (5 ^th Generation Mobile networks (5G) dedicated spectrum enterprise private networks are connected to each other, and the wireless communication module is connected to the artificial intelligence operation auxiliary platform through one of the wireless access points. The intelligent operation assistance method according to claim 4, wherein after the processor of the portable device controls the wireless communication module of the portable device to send the environmental image, the method further comprises: operating with the artificial intelligence The auxiliary platform loads a three-dimensional model database; and the artificial intelligence operation auxiliary platform executes the identification process according to the environment image and the three-dimensional model database to obtain a position information; wherein the artificial intelligence operation auxiliary platform executes the identification process Identify the target object based on the environment image and the location information.

Images