CN114625354A

CN114625354A - System and method for generating HMI graphics

Info

Publication number: CN114625354A
Application number: CN202111175836.1A
Authority: CN
Inventors: 韦德雅达·劳; 金恩德拉·古加利亚
Original assignee: ABB Schweiz AG
Current assignee: ABB Schweiz AG
Priority date: 2020-12-14
Filing date: 2021-10-09
Publication date: 2022-06-14
Also published as: EP4012551A1; US11734803B2; EP4012551B1; US20220188990A1

Abstract

The present disclosure relates to systems and methods for generating HMI graphics. The present disclosure relates to computer-implemented methods and arrangements for generating human-machine interface (HMI) graphics associated with industrial automation systems implementing machine learning. The method includes receiving (S402) a migrating HMI graphic. The migration HMI graphic is obtained from a legacy HMI graphic and the migration HMI graphic includes a plurality of industrial objects. Further, the method includes identifying (S404) a missing industrial object in the above-mentioned migrated HMI graphic using a historical tag data set, the historical tag data set including a plurality of tagged HMI graphics for the industrial object, the industrial object including a graphic object and a data object. Once the missing industrial object is identified, the HMI graphic is migrated to include the identified missing industrial object is corrected (S406), and a corrected HMI graphic is generated (S408). Further, the method includes transmitting (S408) a notification to the computing device indicating that the HMI graphics have been corrected.

Description

System and method for generating HMI graphics

Technical Field

The present disclosure relates generally to industrial automation systems. And more particularly to a system and method for generating human-machine interface (HMI) graphics associated with industrial automation systems implementing machine learning.

Background

Industrial automation systems include a number of sensors, actuators, and controllers for monitoring and controlling operations and processes in the industry. Such controllers include applications with graphical screens for operators to monitor the operation of the industrial automation system. Graphical screens are provided on computing devices using human-machine interfaces (HMIs).

The HMI is a common connection tool for interfacing with the controller. The HMI exchanges information between the operator and the machine to coordinate multiple industrial processes as needed. The goal of HMI design is to enable an operator to manage the manufacturing process through improved decisions while preventing fault conditions to reduce downtime. Thus, using context/context data to perceive changes in information and information flow provides an effective operator HMI design practice. The migration or upgrade strategy of existing HMI designs to the latest version depends on a variety of factors such as improved user interface, added features, and efficient communication. The main challenge herein is to design and debug the target HMI graphics as efficiently and economically as possible.

There are a variety of HMI migration tools; however, these migration tools are not 100% accurate and require human intervention to generate the target HMI graphic. The manual authentication task consumes a lot of time. More specifically, when 300 interventions are required versus 400 HMIs, manual intervention for each HMI can be time consuming and error prone. Furthermore, for complex HMI objects, there are a large number of objects of different sizes and shapes, and manual validation of these objects can be cumbersome, thereby reducing HMI migration efficiency. Accordingly, such manual solutions may not be suitable during HMI migration, and thus there is a need for improved HMI migration arrangements that improve HMI migration efficiency and accuracy.

Disclosure of Invention

The inventors of the present invention, after inventive and insightful reasoning, have realized that there are anticipated problems as more and more HMI graphic transformations need to be implemented in the future market as discussed above and below.

The present disclosure seeks to disclose a system and method to generate and/or convert HMI graphics from one version to another, e.g., from a legacy HMI to a migration HMI, in an efficient/automatic manner to infer and provide for.

The present disclosure provides a solution that seeks to mitigate, alleviate or eliminate one or more of the above-identified deficiencies in the art and disadvantages singly or in any combination and provides a solution for an automatic/efficient mechanism to provide improved HMI graphics during HMI migration.

A first aspect is a computer-implemented method for generating an HMI graphic associated with an industrial automation system. The method includes receiving a migration HMI graphic. The migration HMI graphic is obtained from a legacy HMI graphic and the migration HMI graphic includes a plurality of industrial objects. Further, the method includes identifying a missing industrial object in the migrated HMI graphic using a historical tag data set that includes a plurality of tag HMI graphics for the industrial object, the industrial object including a graphic object and a data object. Once the missing industrial object is identified, the migrated HMI graphic is corrected to include the identified missing industrial object and a corrected HMI graphic is generated and output to the system. In some aspects, a notification indicating that the HMI graphics have been corrected is transmitted to a computing device or any other portable device of an operator or any associated individual operating the industrial operating system.

In some embodiments, the industrial object in the legacy HMI graphic is associated with an older version of the industrial automation system/industrial device, and the industrial object(s) in the migration HMI graphic are associated with a newer version of the industrial automation system/industrial device.

In some embodiments, the at least one missing industrial object indicates that the at least one industrial object is missing in the scene of the migrated HMI graphic (118).

In some embodiments, a method for identifying lost objects in a migration HMI graphic includes identifying industrial objects in both the migration HMI graphic and a legacy HMI graphic using object identification techniques. The method also includes determining a difference between the identified industrial objects in both the migrating HMI graphic and the legacy HMI graphic, and identifying a missing industrial object in the migrating HMI graphic based on the determined difference and using a historical HMI tag data set for at least one industrial object.

In some embodiments, the accuracy of a corrected HMI graphic comprising an industrial object is evaluated using a historical HMI tag data set comprising a plurality of tagged HMI graphics (or images) of the industrial object. In certain aspects, the historical HMI tag data set can be obtained based on domain knowledge/domain experts.

In some embodiments, a method for identifying lost objects in a migrating HMI graphic includes: placeholders in the migration HMI graphics are obtained, locations of missing industrial objects in the migration HMI graphics are detected based on the placeholders, and a bounding box is created around the identified placeholders.

In some embodiments, the bounding box is created using template matching and image processing techniques.

In some embodiments, the method further comprises evaluating the identified lost industrial object using a one-to-one mapping between the plurality of industrial objects. In some aspects, the one-to-one mapping model utilizes image processing techniques in accordance with a historical HMI tag data set that includes a plurality of tagged HMI graphics for an industrial object/device.

A second aspect is an industrial automation system that implements machine learning to generate HMI graphics. The industrial automation system includes a processor, a memory, and a graphical object detection model coupled to the processor and the memory. The graphical object detection model is configured to receive a migration HMI graphic. The migration HMI graphic is obtained from a legacy HMI graphic and the migration HMI graphic includes a plurality of industrial objects. Further, the graphical object detection model is configured to identify a missing industrial object in the aforementioned migrated HMI graphic using a historical tag data set comprising a plurality of tag HMI graphics for industrial objects comprising graphical objects and data objects. Once the missing industrial object is identified, the migrated HMI graphic is corrected to include the identified missing industrial object and a corrected HMI graphic is generated and output. In some aspects, a notification indicating that the HMI graphics have been corrected is also transmitted to the computing device or any other portable device of an operator or any associated individual operating the industrial operating system.

Advantageously, some embodiments provide automatic correction of HMI graphics (i.e., legacy HMI) that otherwise require manual effort to correct the HMI graphics (i.e., migrate the HMI graphics) during the HMI migration process.

Advantageously, some embodiments improve the efficiency and accuracy of HMI migration by implementing machine learning and image processing techniques.

Advantageously, some embodiments leverage industrial domain knowledge and machine vision based intelligence during HMI migration to perform automatic verification and correction of HMI graphics.

Drawings

The foregoing will be apparent from the following more particular description of exemplary embodiments, as illustrated in the accompanying drawings in which like reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating example embodiments.

FIGS. 1A and 1B are block diagrams illustrating embodiments of an industrial automation system;

FIG. 2 is a block diagram illustrating a computing environment implementing the industrial automation system of FIG. 1A;

fig. 3 is an example configuration of a graphical object detection model of the industrial automation system of fig. 1A, in accordance with some example embodiments;

fig. 4 is a flow chart schematically depicting the proposed method according to some example embodiments;

FIG. 5 is a flow diagram schematically depicting a process for identifying and repairing a lost industrial object onto a migration HMI graphic according to some example embodiments; and

6-8 are example scenarios of HMI graphic migration from a legacy HMI graphic to a migrated HMI graphic and further to a corrected HMI graphic in accordance with some example embodiments.

Detailed Description

Aspects of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings. The systems and methods disclosed herein may, however, be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Like numbers refer to like elements throughout.

The terminology used herein is for the purpose of describing particular aspects of the disclosure only and is not intended to be limiting of the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

Some example embodiments presented herein relate to systems and methods for generating HMI graphics associated with an industrial automation system implementing machine learning. As part of the development of the example embodiments presented herein, a problem will first be identified and discussed.

The HMI migration tool is used to convert/migrate HMI graphics from one version to another. However, these HMI migration tools are not 100% accurate and require manual intervention to generate the target HMI graphic. In addition, the target human machine interface graphics are manually corrected to make them look aesthetically pleasing and have proper alignment, including each graphic element and its resolution, maintaining accurate translation of static and dynamic information, and so on. Such formatted HMI require manual review and correction of any discrepancies, which can reduce HMI migration efficiency.

Advantageously, some embodiments are directed to improving efficiency and accuracy of HMI graphic migration by enabling automated intervention to migrate HMI graphics that automatically verifies and corrects differences in HMI graphics.

Advantageously, some embodiments provide sufficient support for migration of complex legacy graphical objects, reduced migration effort and cost, more automation and less manual engineering effort, and improved consistency between HMI graphics.

Advantageously, some embodiments enable automatic validation and correction of HMI graphics using industry domain knowledge during HMI translation and machine vision based intelligence.

Advantageously, some embodiments provide a reference platform for comparing HMI migration tools. That is, a build of comparison functionality between any two arbitrary HMI migration tools is provided.

As shown in fig. 1, an industrial automation system 100 is used to control a plurality of industrial devices/components that are responsible for handling different processes and machines in a plurality of industrial plants (or any technical installation). For example, process control in manufacturing products (i.e., insulators in insulator manufacturing plants) requires continuous monitoring of industrial equipment/components (e.g., pumps, valves, motors, boilers, turbines, and generators (BTGs)) that are required to be used in processing raw materials (e.g., clay, porcelain, etc.) in the process of manufacturing insulators. The industrial automation system 100 facilitates such continuous monitoring of industrial devices/components in such multiple industrial plants. Each industrial plant represents/indicates one or more manufacturing facilities for producing at least one product (i.e., insulator) or other material.

The industrial automation system 100 includes a plurality of industrial devices 102 (or industrial objects) responsible for controlling/monitoring and/or manufacturing a product (e.g., insulator) or material. The industrial automation system 100 also includes one or more sensors 104a and one or more actuators 104 b. The sensors 104a and actuators 104b are communicatively coupled to the plurality of industrial devices 102. In some aspects, the sensors 104a can measure/monitor configuration and/or operational readings of the industrial device 102. Configuration and/or operational readings (e.g., temperature, pressure, current flow rate), and further triggers an alarm in the event of any anomaly when such configuration and/or operational readings are read/monitored. In some aspects, the actuator 104b is responsible for driving the operational functions of the industrial equipment 102. For example, the actuator 104b may be responsible for closing the valve in the event that the sensor 104a issues an alarm. Accordingly, each actuator 104b is responsible for one or more conditions of operating the industrial device 102 in the industrial automation system 100.

In some aspects, the connections between the sensors 104 a/actuators 104b and the plurality of industrial devices 102 are via a network interface 106, such as an ethernet network, an electrical signal network, Bluetooth, or any other network or type(s) of network. In some other aspects, the network interface 106 between the sensor 104a and the actuator 104b facilitates the transfer of data between the sensor 104a and the actuator 104 b.

All measurements from the sensors 104a and actuators 104b are communicated to the industrial controller 108 via the network interface 106. In some aspects, the industrial controller 108 can include, for example, a Distributed Control System (DCS), a Programmable Logic Controller (PLC), a proportional-integral-derivative (PID) controller, or the like. In one example, each controller 108 may also represent a computing device. The controller 108 is responsible for controlling the operation of the sensors 104a and actuators 104 b. For example, the controller 108 may utilize data acquired from the sensors 104a to operate the actuators 104 b. In some other aspects, the controller 108 may be used to control/create/modify the configuration of the sensors 104a and actuators 104 b. In some other aspects, the controller 108 may be configured to create and control logical connections between each sensor 104a and between each actuator 104 b. Further, the controller 108 can control the configuration and/or connection between each sensor 104 a/actuator 104b and the industrial assembly 102.

Each controller 108 may also be responsible for controlling one or more aspects of the industrial process. For example, controlling the operation of a boiler for treating raw materials, such as clay used for manufacturing insulators.

An operator of the industrial automation system 100 can access and interact with the controller 108 using a Human Machine Interface (HMI) 110. In some aspects, the controller 108 may have an HMI embedded/installed (not shown) within the controller 108. Access and interaction with the controller 108 may be performed via the computing device 112. In some aspects, access to the controller 108 may include, for example, access to the connections and configurations of the sensors 104a, actuators 104b, and some industrial equipment 102. Further, in some aspects, interaction with the controller 108 can include, for example, identifying different states of the industrial process, such as alarm states, operational values of various industrial devices 102, and so forth.

In some aspects, the computing device 112 may comprise, for example, any computer, such as a stand-alone computer, a laptop computer, a Personal Computer (PC), a plurality of display screens, a tablet computer, a portable device, and the like. The computing device 112 can include a display unit and a Graphics Processing Unit (GPU) configured to process and display graphical images (e.g., HMI graphics 114) of the industrial device 102 and connections between each industrial component 102. In one example, the HMI graphic 114 indicates a graphical form of connection between each industrial object 102 and each sensor 104 a/actuator 104b, also displayed by the computing device 112. Further, the HMI graphics 114 indicate process lines representing process connections, meter lines indicating process recording or measurement points, control connections, and the like. Each industrial component 102 can be uniquely represented/graphically displayed onto a display unit of the computing device 112 using the HMI. For example, each actuator 104b may be represented graphically as a valve, while each sensor 104a may be represented as a meter (as shown in FIGS. 7A-7C).

As described above, each industrial component 102 can be uniquely represented/graphically displayed onto a display unit of the computing device 112, and thus similarly, each controller 108 (or each version of operating firmware of the controller 108) includes its own HMI graphics library that is different from the operating firmware of the other controllers or a version of the other controllers. Thus, whenever a version of a controller 108 migrates to other controllers or to a newer/updated version of operating firmware, migration of HMI graphics is always required. In some aspects, an HMI graphic conversion model/HMI migration tool 116 can be used to convert or migrate a legacy HMI graphic 114 to a new HMI graphic (i.e., migrate HMI graphic 118, as shown in FIG. 1B) to provide an enhanced user interface experience.

The migration or upgrade strategy that the legacy HMI graphic 114 is designed to migrate to the migrating HMI graphic 118 depends on a variety of factors such as improved user interface, added features, and efficient communication. The main challenge herein is to design and debug the target HMI graphics (final HMI graphics) as efficiently and economically as possible. The existing/legacy HMI graphics translation tool migrates graphics data from one HMI to another. However, due to the dynamic nature of the functional specification in the newly built application, existing HMI graphic migration tools may not possess all of the functionality to migrate from a legacy HMI graphic to a new HMI graphic. Most HMI graphic migration tools remain inaccurate due to incorrect symbol or tag mappings. Therefore, manual intervention is required to fix the inaccuracy (which increases the chance of error) when formatting the target HMI graphic. Such manual work can be expensive and does not allow the user to obtain the full benefits of automatic switching.

Unlike conventional mechanisms for migrating HMI graphics, the present invention seeks to implement/incorporate a machine learning mechanism to automatically identify inaccuracies (i.e., missing industrial graphic objects) in migrating HMI graphics during HMI conversion.

To achieve the desired technical effect of the present invention, namely, to automatically identify inaccuracies in migrating human-machine interface graphics during HMI conversion, the present invention seeks to provide a graphical object detection model 120 (interchangeably used as HMI graphical object detection model 120) configured to perform the proposed method. In some aspects, the graphical object detection model 120 is configured to obtain the migration HMI graphics 118 (an output of the HMI migration tool 116). The graphical object detection model 120 is then configured to automatically identify inaccuracies in the migration HMI graphic 118 and to correct the inaccuracies in the migration HMI graphic 118. Once corrected, the graphical object detection model 120 can be configured to output the corrected HMI graphic 122 (i.e., including all industrial graphical objects and textual information present in the legacy HMI graphic 114), as shown in FIG. 1B, thereby improving the accuracy and efficiency of HMI migration.

In some aspects, the graphical object detection model 120 may be hosted on a cloud computing server and may be accessed by an operator/person using any suitable network interface. In some other aspects, the graphical object detection model 120 may be part of the HMI migration tool 116.

In some aspects, the present invention provides a reference platform for comparing an HMI conversion tool (i.e., HMI migration tool 116) (integrated with graphical object detection model 120) to any other HMI migration tool (not shown). For example, a comparison function between any two arbitrary HMI conversion tools is constructed by: a) utilizing the HMI conversion tool 116 to identify missing graphical objects and textual information on the migrated HMI graphic 118, b) repeating step a) using other HMI migration tools. Further, the missing information is checked (at step c)) on a display unit of the computing device 112 to compare the HMI migration tool 116 with other migration tools. The dashboard gives a detailed analysis of the computing device 112 to display legacy HMI graphics 114 from the legacy system, and the option to select between the two HMI migration tools. The output of the comparison after performing steps a, b, and c above is displayed on a display unit of the computing device 112. Thus, the reference platform indicates comparisons between the HMI migration tools that can be effectively used by developers, designers, and operators to analyze and compare across HMI migration tools.

Referring to fig. 2, computing environment 200 may include a processing unit 201, one or more memory devices 202 (referred to herein as memory 202), a storage unit 203, an input unit 204, and an output unit 205. The computing environment 200 may also include one or more buses 206, the buses 206 functionally coupling the various components of the computing environment 200.

The memory 202 may include volatile memory such as Random Access Memory (RAM) (memory that maintains its state when powered) and/or non-volatile memory such as Read Only Memory (ROM), flash memory, ferroelectric RAM (fram) (memory that maintains its state even when not powered). Persistent data storage (as that term is used herein) may include non-volatile memory. In some example embodiments, volatile memory may enable faster read/write access than non-volatile memory. However, in certain other example embodiments, certain types of non-volatile memory (e.g., FRAM) may enable faster read/write access than certain types of volatile memory. In some examples, storage unit 203 can be identical to memory 202. In various implementations, the memory 202 may include a variety of different types of memory, such as various types of Static Random Access Memory (SRAM), various types of Dynamic Random Access Memory (DRAM), various types of non-alterable ROM, and/or writable variants of ROM, such as electrically erasable programmable read-only memory (EEPROM), flash memory, and so forth. The memory 202 may include main memory as well as various forms of cache memory such as instruction cache(s), data cache(s), and translation lookaside buffer(s) (TLB), among others. Further, a cache such as a data cache may be a multi-level cache organized as a hierarchy of one or more cache levels (L1, L2, etc.).

Storage unit 203 may include removable storage and/or non-removable storage, including but not limited to magnetic storage, optical storage, and/or tape storage. Storage unit 203 may provide non-volatile storage of computer-executable instructions and other data. The removable and/or non-removable storage unit 203 and the memory 202 are examples of a computer-readable storage medium (CRSM).

In some aspects, the storage unit 203 stores a historical HMI tag data set 207. The historical HMI tag data set 207 includes a plurality of tagged HMI graphics for the industrial object 102. The tagged HMI graphics are based on industrial domain knowledge (obtained from various sources, e.g., portals of industrial domain experts, online industrial domain portals, industry specific domain portals, etc.). For example, the industrial domain knowledge can include all information about missing graphical objects in the migration HMI graphic 114 and their corresponding regions on the migration HMI graphic 114.

Accordingly, the present invention utilizes industrial domain knowledge (i.e., historical HMI tag data set 207) during HMI graphic conversion and machine learning (or machine vision) based intelligence to perform automatic verification and correction of HMI graphics.

Storage unit 203 may store computer-executable code, instructions, or the like, that may be loaded into memory 202 and executed by processing unit 201 to cause processing unit 201 to perform or initiate various operations. The storage unit 203 may additionally store data that may be copied to the memory 202 for use by the processing unit 201 during execution of the computer-executable instructions. Furthermore, output data generated as a result of execution of computer-executable instructions by processing unit 201 may be initially stored in memory 202 and may ultimately be copied to storage unit 203 for non-volatile storage.

More specifically, the storage unit 203 may store an operating system (O/S); a database 102 configured to access a memory 202; and one or more program modules, applications, engines, managers, computer-executable code, scripts, etc., such as the various modules of the graphical object detection model 120. Any components described as being stored in storage unit 203 may include any combination of software, firmware, and/or hardware. The software and/or firmware may include computer-executable instructions (e.g., computer-executable program code) that may be loaded into memory 202 for execution by one or more of processing units 201 to perform any of the corresponding operations previously described.

The processing unit 201 may be configured to access the memory 202 and execute the computer-executable instructions loaded therein. For example, the processing unit 201 may be configured to execute computer-executable instructions of various program modules, applications, engines, managers, and/or the like of the graphical object detection model 120 to cause or facilitate various operations to be performed in accordance with one or more embodiments of the present disclosure. Processing unit 201 may include any suitable processing unit capable of accepting data as input, processing the input data according to stored computer-executable instructions, and generating output data. Processing unit 201 may include any type of suitable processing unit, including but not limited to a central processing unit, microprocessor, Reduced Instruction Set Computer (RISC) microprocessor, Complex Instruction Set Computer (CISC) microprocessor, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), system on chip (SoC), Digital Signal Processor (DSP), or the like. Furthermore, processing unit 201 may have any suitable micro-architectural design that includes any number of constituent components, such as registers, multiplexers, arithmetic logic units, cache controllers for controlling read/write operations to cache memory, branch predictors, and the like. The micro-architectural design of processing unit 201 may be capable of supporting any of a variety of instruction sets.

The input unit 204 and the output unit 205 may facilitate the reception of input information by the graphical object detection model 120 from one or more I/O devices and the output of information from the graphical object detection model 120 to one or more I/O devices. The I/O device may include any of a variety of components, such as a display or display screen with a touch surface or touch screen; an audio output device, such as a speaker, for producing sound; an audio capture device, such as a microphone; an image and/or video capture device, such as a camera; a haptic unit; and so on. Any of these components may be integrated into components of the computing environment 200 or only into the industrial automation system 100, or may be separate. The I/O devices may also include, for example, any number of peripheral devices, such as data storage devices, printing devices, and the like. The input unit 204 and the output unit 205 may also include I/O interfaces for external peripheral device connections, such as Universal Serial Bus (USB), firewire, Thunderbolt, ethernet ports, or other connection protocols that may connect to one or more networks. .

The I/O interface(s) may also include one or more connection ports to connect to one or more controllers 108, interfaces for network cabling, and to other computing devices (similar to computing device 112). The connection to the port may be through an electrical signal cable, HMI cable, or the like. Further, the I/O interface(s) may also include one or more antennas to connect to one or more networks through a Local Area Network (LAN) or, alternatively, through a Wireless Local Area Network (WLAN) (e.g., Wi-Fi) wireless, Bluetooth, and/or wireless network radio.

Bus(s) 206 may include at least one of a system bus, a memory bus, an address bus, or a message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between various components of visual perception module 103. Bus(s) 206 may include, but are not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and the like.

Referring to fig. 3, the graphical object detection model 120 may include an image processing unit 302, a text mining unit 304, a Machine Learning (ML) unit 306, and a graphical object generation unit 308.

The graphical object detection model 120 may be configured to receive the migrated HMI graphics 118 from the HMI migration tool 116 using the image processing unit 302. The migration HMI graphic 118 is obtained from the legacy HMI graphic 114. That is, the migration HMI graphic 118 is an output HMI graphic from the HMI migration tool 116 (responsible for migrating the legacy HMI graphic 114). In some aspects, the migration HMI graphics 118 include one or more industrial objects 102 in the form of graphical objects or data objects. The graphical objects may include, for example, graphical representations of the sensors 104a, the actuators 104b, the industrial objects 102 (e.g., boilers, pumps, etc.), and the connectors (or indicators, such as arrow indicators) between the industrial objects 102 and between the sensors 104a and the actuators 104 b. Further, the data objects may include, for example, text or any number associated with each graphical representation of the sensors 104a, actuators 104b, and industrial object 102. This is illustrated, for example, in the form of the legacy HMI graphic 114 and the corrected HMI graphic 122.

Further, the HMI graphic object detection model 120 can be configured to identify one or more missing industrial objects in the migrated HMI graphic 118. The components of the graphical object detection model 120 (e.g., the image processing unit 302, the text mining unit 304, and the Machine Learning (ML) unit 306) communicate with the storage unit 203 to utilize the historical HMI marker data set 207 during identification of lost industrial objects in the aforementioned migrated HMI graphics 118.

In some aspects, the image processing unit 302 may be configured to identify the lost industrial object and the corresponding location of the lost industrial object in the migrated HMI graphic 118 based on an object recognition technique that identifies HMI graphic objects in both the legacy HMI graphic 114 and the migrated HMI graphic 118. In accordance with an aspect of the subject invention, to find a missing industrial object in the migrating HMI graphic 118, the image processing unit 302 can be configured to use an image map to determine a difference between the identified industrial objects in the legacy HMI graphic 114 and the migrating HMI graphic 118.

In certain aspects, the location of the missing industrial object in the migration HMI graphic 118 can be obtained by detecting the presence of a placeholder/default object in the migration HMI graphic 118. The placeholder/default object in the migration HMI graphic 118 is acquired during the translation of the legacy HMI graphic 114. That is, when legacy HMI graphics 114 are translated using HMI migration tool 116, the translated HMI graphics (i.e., migrated HMI graphics 118) include one or more placeholders/default objects that indicate missing industrial objects.

In some aspects, the image processing unit 302 may be configured to use a combination of template matching and image processing methods/models to obtain a bounding box around the placeholder/default object. The bounding box indicates the location of the missing industrial object in the migration HMI graphic 118 and can be used as an input to the ML unit 306. For example, bounding boxes are placed around placeholders and these are potential areas (inputs) for ML cell 306 to identify missing industrial objects. In some aspects, pixel inputs associated with the legacy HMI graphics 114 in the area encompassed by the bounding box are obtained by the ML unit 306 for detecting possible industrial objects lost in the migrated HMI graphics 118.

In some aspects, the ML unit 306 implements a Convolutional Neural Network (CNN) model to identify relationships between industrial objects 102 of the industrial automation system 100 using the historical HMI tag data set 207. The image processing unit 302 communicates with an ML unit 306 implementing NN. The ML unit 306 can be configured to classify the industrial object 102 (identified from the image processing unit 302) in both the legacy HMI graphic 114 and the migration HMI graphic 118 using the historical HMI tag data set 207. In one example, semantic information for the industrial objects 102 is determined from the historical HMI tag data set 207, wherein the semantic information includes relationships between the industrial objects 102, contextual connections between the industrial objects 102, and the like. In some embodiments, ML unit 306, which implements the CNN model, is trained using historical HMI tag data sets 207. In some other embodiments, ML unit 306 implementing the CNN model may be trained using historical data of the automated HMI objects with associated tags and cross-trained using historical tag data associated with the automated HMI objects. For example, tag data associated with the automation HMI object can also be generated by user input based on at least one feedback stimulus. In yet another embodiment, ML unit 306 implementing the CNN model may be trained using unlabeled historical data of the automated HMI object and cross-trained using labeled historical data associated with the automated HMI object.

In addition to semantic mapping of the industrial object 102 and its corresponding region/location on the migrated HMI graphic 118, the HMI graphic object detection model 120 utilizes a one-to-one mapping to identify the correct industrial object to place in the migrated HMI graphic 118. That is, the HMI graphic object detection model 120 replaces placeholders (indicated by bounding boxes) with the correct industrial objects in the migrated HMI graphics 118.

Thus, missing industrial objects in the designated area of the migrated HMI graphic 118 can be automatically recovered, thereby reducing manual effort or human intervention. Therefore, a large amount of time can be reduced by reducing manual work.

In some aspects, the text-mining unit 304 may be configured to identify missing text or data content and its bounding boxes in the migration HMI graphic 118. For example, the text-mining unit 304 implements a scene text recognition technique/model to identify missing text or data content in the migration HMI graphic 118.

In view of the foregoing, FIG. 4 illustrates an example method 400 implemented by the HMI graphical object detection model 120. According to the method 400, the migration HMI graphic 118 is obtained from a legacy HMI graphic 114 (as detailed above). The HMI graphical object detection model 120 then receives the migrated HMI graphic 118 (block S402). The migration HMI graphic 118 includes at least one industrial object.

According to the method 400, at least one missing industrial object is identified in the migration HMI graphic 118 using the historical HMI tag data set 207 for the at least one industrial object (block S404). The at least one industrial object includes a graphical object and a data/text object.

According to the method 400, the migration HMI graphic 118 is corrected to include at least one missing industrial object (block S406). Further, according to the method 400, a corrected HMI graphic is generated wherein the corrected HMI graphic 122 merges at least one missing industrial object (block S408).

According to the method 400, a notification is transmitted to the computing device 112 indicating that the HMI graphics 122 have been corrected (block S410). The notification may include, for example, an alarm configured by the computing device 112 or a user desired configuration, an audible notification such as a beep, or the like.

In view of the foregoing, FIG. 5 illustrates an example method 500 implemented by the HMI graphic object detection model 120 to identify and repair lost industrial objects in a migrated HMI graphic 118. According to the method 500, at least one industrial object is identified 102 in the above-described migration HMI graphic 118 and legacy HMI graphic 114 (block S502). An object recognition model can be used for such identification of the industrial object 102.

Further, according to the method 500, a difference between the identified industrial objects in the legacy HMI graphic 114 and the migrating HMI graphic 118 is determined (block S504).

Further, according to the method 500, the presence of placeholder/default objects in the migration HMI graphics 118 is identified (block S506). The placeholder/default object may be used to identify the location of the missing industrial object in the migration HMI graphic 118. In certain aspects, the placeholder/default object is obtained directly from the migration HMI graphic 118, the migration HMI graphic 118 also indicating that one or more industrial objects are lost (in a particular region) as compared to one or more industrial objects in the legacy HMI graphic 114.

Further, according to the method 500, at least one bounding box is created around the identified at least one placeholder (block S508). To obtain the bounding box for the default object or placeholder, the HMI graphical object detection model 120 utilizes a combination of template matching and image processing techniques.

Further, according to the method 500, information regarding both the lost industrial object and its corresponding region is identified using a bounding box on the migration HMI graphic 118 (block S510). This can be accomplished by using ML unit 306, which ML unit 306 utilizes a one-to-one mapping to identify (using historical HMI tag data sets 207) the correct industrial object to place in the migration HMI graphic 118.

Further, according to the method 500, the placeholders are replaced with the correct industrial object on the migration HMI graphic 118 (block S512).

Referring to fig. 6A, the legacy HMI graphic 114 illustrates a graphical form of an industrial object (i.e., the main pump 602) on the computing device 112 of the industrial automation system 100. The legacy HMI graphic 114 also illustrates graphical forms of other industrial objects including process lines, connecting lines, and any textual data associated with an industrial object on the computing device 112 of the industrial automation system 100.

In some aspects, the legacy HMI graphics 114 are transferred (as input) to the HMI migration tool 116 for HMI migration purposes. The output of HMI migration tool 116 can be represented as a migration HMI graphic 118 (shown in FIG. 6B).

Referring to FIG. 6B, the migration HMI graphic 118 illustrates a migration graphical form of the industrial object 102. As described above, these HMI migration tools are not 100% accurate and require human intervention to generate the target HMI graphics. For example, as shown in FIG. 6B, in the migration HMI graphic 118, an industrial object (i.e., the primary pump 602) is lost. Thus, according to a legacy approach, a manual check is performed on the migration HMI graphics 118 to identify a missing industrial object, i.e., the primary pump 602. Such manual review/inspection of the formatted HMI, as well as further manual correction of discrepancies (lost industrial objects), reduces the efficiency of HMI migration.

Unlike such conventional approaches, the proposed method utilizes historical HMI tag images (of industrial objects) to train the HMI graphics object detection model 120 using machine learning techniques to automatically verify and correct any discrepancies in the migrated HMI graphics 118. Furthermore, the proposed method also utilizes an object recognition model to resolve conflicts resulting from migrating fuzzy or nearly similar looking industrial objects in the HMI graphics 118.

Referring again to FIG. 6B, the HMI graphical object detection model 120 creates a bounding box 604 to replace the default object/placeholder to indicate a missing primary pump 602. The HMI graphical object detection model 120 then performs a one-to-one mapping to identify the correct object(s) (including the plurality of HMI tag images of the industrial object) to be placed in the migrated HMI graphic 118 using machine learning techniques and the historical HMI tag data set 207. Placeholders herein indicate locations where industrial objects are lost.

Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces/restores the placeholder on the migration HMI graphic 118 with the correct industrial object. For example, referring to FIG. 6C, missing primary pump 606 replaces the placeholder/default object of bounding box 602. The corrected HMI graphic 122 is then transmitted for further addition/review (to meet additional customer requirements) and constitutes a target HMI graphic.

Similar to fig. 6A, fig. 7B illustrates a legacy HMI graphic 114 in graphical form comprising an industrial object (i.e., a tank 702) on a computing device 112 of the industrial automation system 100. The legacy HMI graphic 114 also illustrates graphical forms of other industrial objects including arrowed process lines 704 and 706 (down), process lines, connecting lines, and any textual data associated with the industrial object on the computing device 112 of the industrial automation system 100.

Legacy HMI graphics 114 are transferred (as input) to HMI migration tool 116 for HMI migration purposes. Output from the HMI migration tool 116 can be represented as a migration HMI graphic 118 (shown in FIG. 7B).

Referring to FIG. 7B, the migration HMI graphic 118 illustrates a migration graphical form of the industrial object 102. As described above, the industrial object (i.e., the tank 702) is lost and there is a difference in migrating the arrowed process lines 704 and 706 (up) in the HMI graphic 118.

As described above, the HMI graphical object detection model 120 creates bounding

boxes

708 and 710 to replace default objects/placeholders to indicate the differences in the lost canister 702 and the

arrowed process line

704 and 706. The HMI graphical object detection model 120 then performs a one-to-one mapping to identify the correct object(s) (including the plurality of HMI tag images of the industrial object) to be placed in the migrated HMI graphic 118 using machine learning techniques and the historical HMI tag data set 207.

Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces/restores the placeholder on the migration HMI graphic 118 with the correct industrial object. For example, referring to FIG. 7C, the missing canister 702 and the differences in

arrowed process line

704 and 706 replace the placeholder/default objects of bounding

boxes

708 and 710. The corrected HMI graphic 122 is then transmitted for further addition/review (to meet additional customer requirements) and to compose a target HMI graphic.

Similar operations performed in FIGS. 6A-6C are repeated to correct the HMI graphics shown in FIGS. 8A-8C. Referring to fig. 8A, a legacy HMI graphic 114 is displayed on the computing device 112 of the industrial automation system 100, the legacy HMI graphic 114 including a graphical form of the industrial objects (i.e., the CONTROL valves 802& KV-1804A and mol. sieve CONTROL), the graphical form having text buttons and values thereof.

Legacy HMI graphics 114 are transferred (as input) to HMI migration tool 116 for HMI migration purposes. The output of the HMI migration tool 116 can be represented as a migration HMI graphic 118 (shown in FIG. 8B).

Referring to FIG. 8B, the migration HMI graphic 118 illustrates a migration graphical form of the control valve 802 and the text button along with its value is missing in the migration HMI graphic 118.

As described above, the HMI graphical object detection model 120 creates the bounding box 804 to replace the default object/placeholder to indicate the missing control valve 802 and text button and their values in the migration HMI graphics 118. The HMI graphical object detection model 120 then performs a one-to-one mapping to identify the correct object to place in the migrated HMI graphic 118 using machine learning techniques and the historical HMI tag data set 207 (comprising a plurality of HMI tag images of the industrial object).

Once the correct industrial object to be placed on the migration HMI graphic 118 is identified, the HMI graphic object detection model 120 replaces/restores the placeholder on the migration HMI graphic 118 with the correct industrial object. For example, referring to FIG. 8C, the missing control valve 802 and text button, along with their values, replace the placeholder/default object of the bounding box 804. The corrected HMI graphic 122 is then transmitted for further addition/review (to meet additional customer requirements) and constitutes a target HMI graphic.

Thus, as discussed above in FIGS. 6-8, the present invention provides automatic verification and correction of any discrepancies in the migrated HMI graphics 118, which significantly saves production/developer time by reducing the manual effort to correct HMI graphics.

Aspects of the disclosure are described with reference to the accompanying figures (e.g., block diagrams and/or flowcharts). It is to be understood that the entities of the figures (e.g., blocks of block diagrams), and combinations of entities in the figures, can be implemented by computer program instructions, which can be stored in a computer-readable memory and loaded onto a computer or other programmable data processing apparatus. Such computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, and/or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer and/or other programmable data processing apparatus, create means for implementing the functions/acts specified in the block diagrams and/or flowchart block or blocks.

In some implementations and in accordance with some aspects of the present disclosure, the functions or steps noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Further, according to some aspects of the present disclosure, the functions or steps noted in the blocks may be performed continuously in a loop.

In the drawings and specification, there have been disclosed exemplary aspects of the disclosure. However, many variations and modifications may be made to these aspects without substantially departing from the principles of the present disclosure. Accordingly, the present disclosure is to be considered as illustrative and not restrictive, and not limited to the particular aspects discussed above. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

The description of the example embodiments provided herein is presented for purposes of illustration. It is not intended to be exhaustive or to limit the example embodiments to the precise form disclosed, and modifications and variations are possible in light of the above teachings or may be acquired from practice of various alternatives to the provided embodiments. The examples discussed herein were chosen and described in order to explain the principles and the nature of various example embodiments and its practical application to enable one skilled in the art to utilize the example embodiments in various ways and with various modifications as are suited to the particular use contemplated. The features of the embodiments described herein may be combined in all possible combinations of methods, apparatus, modules, systems, and computer program products. It should be understood that the example embodiments presented herein may be practiced in any combination with each other.

It should be noted that the word "comprising" does not necessarily exclude the presence of other elements or steps than those listed and the word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. It should also be noted that any reference signs do not limit the scope of the claims, that the exemplary embodiments may be implemented at least in part by means of hardware and software, and that several "means", "units" or "devices" may be represented by the same item of hardware.

Various example embodiments described herein are described in the general context of method steps or processes, which may be implemented in one aspect by a computer program product, embodied in a computer-readable medium, including computer-executable instructions, such as program code, executed by computers in network environments. Computer-readable media may include removable and non-removable storage devices, including but not limited to Read Only Memory (ROM), Random Access Memory (RAM), Compact Disks (CDs), Digital Versatile Disks (DVDs), and the like. Generally, program modules may include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Computer-executable instructions, associated data structures, and program modules represent examples of program code for executing steps of the methods disclosed herein. The particular sequence of such executable instructions or associated data structures represents examples of corresponding acts for implementing the functions described in such steps or processes.

In the drawings and specification, there have been disclosed exemplary embodiments. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation, the scope of the embodiments being defined by the following claims.

Claims

1. A computer-implemented method for generating human-machine interface, HMI, graphics associated with an industrial automation system (100) implementing machine learning, the method performed by a graphical object detection model (120), the method comprising:

receiving (S402) a migration HMI graphic (118), wherein the migration HMI graphic (118) is obtained from an old HMI graphic (114), and wherein the migration HMI graphic (118) comprises a plurality of industrial objects (102);

identifying (S404) at least one missing industrial object in the migrated HMI graphic (118) using a historical HMI tag data set (207) of at least one industrial object (102), wherein the at least one industrial object (102) comprises a graphic object and a data object;

correcting (S406) the migrated HMI graphic (118) to include the at least one missing industrial object;

generating (S408) a corrected HMI graphic (122) comprising the at least one missing industrial object; and

transmitting (S410) a notification to a computing device indicating the corrected HMI graphics (122).

2. The computer-implemented method of claim 1, wherein identifying (S404) the at least one missing object in the migrating HMI graphic (118) comprises:

identifying (S502) the at least one industrial object (102) in the migrated HMI graphic (118) and the legacy HMI graphic (114) using an object recognition technique;

determining (S504) a difference between the identified industrial object in the legacy HMI graphic (114) and the identified industrial object in the migration HMI graphic (118); and

identifying (S506) the at least one missing object in the migrated HMI graphic (118) based on the determined difference and using the historical HMI markup data set (207) of the at least one industrial object (102).

3. The computer-implemented method of claims 1 and 2, wherein identifying the at least one missing industrial object in the migration HMI graphic (118) comprises:

obtaining at least one placeholder in the migration HMI graphic (118);

detecting a location of the at least one missing industrial object in the migration HMI graphic (118) based on the at least one placeholder; and

creating (S508) at least one bounding box around the identified at least one placeholder.

4. The computer-implemented method of claim 3, wherein the at least one bounding box is created using template matching and image processing.

5. The computer-implemented method of claims 1 to 3, further comprising:

evaluating (S510) the identified at least one missing industrial object using a one-to-one mapping between the plurality of industrial objects (102); and

including (S510) the at least one missing industrial object on the at least one respective placeholder of the migrating HMI graphic (118).

6. The computer-implemented method of claims 1-3 and 5, wherein scene text recognition techniques are used to identify the missing industrial object.

7. The computer-implemented method of claim 1, wherein the notification comprises at least one of a textual notification, an audible notification, and a graphical notification, and wherein the historical HMI tag data set (207) comprises a plurality of HMI tag images for the at least one industrial object (102).

8. The computer-implemented method of claim 1, wherein the at least one missing industrial object indicates that at least one industrial object is missing in a scene of the migrated HMI graphic (118).

9. An industrial automation system (100) implementing machine learning to generate human-machine interface (HMI) graphics, wherein the industrial automation system (100) comprises:

a processing unit (201);

a memory (202) communicatively coupled to the processing unit (201); and

a graphical object detection model (120) communicatively coupled to the processing unit (201) and the memory (202), the graphical object detection model (120) configured to:

receiving a migration HMI graphic (118), wherein the migration HMI graphic (118) is obtained from an old HMI graphic (114) and wherein the migration HMI graphic (118) includes at least one industrial object (102),

identifying at least one missing industrial object in the migrated HMI graphic (118) using a historical HMI tag data set (207) for the at least one industrial object (102), wherein the at least one industrial object (102) comprises a graphic object and a data object,

correcting the migrated HMI graphic (118) to include the at least one missing industrial object,

generating corrected HMI graphics (122) comprising the at least one missing industrial object, and

transmitting a notification to a computing device indicating the corrected HMI graphics (122).

10. The industrial automation system (100) of claim 9, wherein the graphical object detection model (120) configured to identify the at least one missing object in the migrated HMI graphic (118) comprises:

identifying the at least one industrial object (102) in the migrated HMI graphic (118) and the legacy HMI graphic (114) using an object recognition technique;

determining a difference between the identified industrial object in the legacy HMI graphic (114) and the identified industrial object in the migration HMI graphic (118); and

identifying the at least one missing object in the migrated HMI graphic (118) based on the determined difference and using the historical HMI tag data set (207) of the at least one industrial object.

11. The industrial automation system (100) of claims 9 and 10, wherein the graphical object detection model (120) configured to identify the at least one missing object in the migrated HMI graphic (118) comprises:

obtaining at least one placeholder in the migration HMI graphic (118);

detecting a location of the at least one lost industrial object in the migrated HMI graphic (118) based on the at least one placeholder; and

creating at least one bounding box around the identified at least one placeholder.

12. The industrial automation system (100) of claims 9-11, wherein the graphical object detection model (120) is further configured to:

evaluating the identified at least one missing industrial object using a one-to-one mapping; and

including the at least one missing industrial object on the respective at least one placeholder of the migrating HMI graphic (118).

13. The industrial automation system (100) of claim 12, wherein scene text recognition techniques are used to identify the missing industrial object.

14. The industrial automation system (100) of claim 9, wherein the notification includes at least one of a textual notification, an audible notification, and a graphical notification, and wherein the historical HMI tag data set (207) includes a plurality of HMI tag images of the at least one industrial object (102).

15. A computer program product comprising a non-transitory computer readable medium having thereon a computer program comprising program instructions, the computer program being loadable into a graphical object detection model (120) and configured to cause execution of the method according to any of claims 1 to 8 when the computer program is run by the graphical object detection model (120).