Interacting with Intelligent Digital Twins

Driven by the trend of Internet of Things (IoT), industrial big data, and Industry 4.0, the concept of smart manufacturing will continue to grow, and new technologies and methods will emerge in the manufacturing intelligent systems domain. One of intelligent technologies that have been recently introduced into industry is known as ‘digital twin’. This proposed project is linked with a semi-conductor manufacturing facility. The semiconductor manufacturing settings generally include a big number of operations and process. This is accompanied by the installation of data collection sensors located in the entire manufacturing facilities and machines. As a result, valuable data, that includes different measurements and different parameters readings, can be logged. Considering the massive volumes of the acquired data, it is becoming more challenging to keep using the traditional data analyses and visualisation tools.

Digital Twin (DT) is based on the concept of cypher-physical simulations (Lin et al. 2018). The main idea of a DT technology contains a combination of physical production process, virtual designed of complex assets, and data flow from both the physical and the virtual systems (Tao et al. 2018). Thus, as a result, there is a need to achieve a high level of convergence and synchronization. Semiconductor manufacturing process generates a massive amount of data. Such data can be effectively invested and analysed to determine some hidden insights. DT promises to visualize, control, and use all of the collected data and create smooth interaction between users and the manufacturing processes. Thus, DT will make the production processes designs more visualized (Tao et al. 2018). It can be used to collect haptic feedback from users to improve the user experience while using the systems. Thus, a unique real-time synchronized communication can be achieved between users and designed digital replica of any physical manufacturing process (El Saddik 2018).

In this project, we seek to design an Intelligent DT (IDT) system within a semiconductor manufacturing to be able to provide a nearly natural interaction to allow users controlling and monitoring the entire factory, in real-time. Thus, it will make the company ready for effective decision-making purposes. The designed interaction of the developed IDT system will consist of the following key stages; (1) design of an advanced virtual environment using a gaming engine. (2) use voice recognition interaction to interact with a virtual assistant in order to discuss real-time scenarios and help take better decisions. (3) use leap motion and Virtual Reality controllers to interact with virtual products and help improve their design without interrupting the physical system. Figure 1 shows the main designed key interaction stages of the IDT system.

(Source: Self)

Interaction key stages of the IDT system

Full size image

This paper is structured as follows. Section 1, Introduction the topic of the paper. Section 2 provided literature review about the concept of DT technology in the semiconductor industry, the concept of human computer interaction in VR and DT and the concept of user experience in VR and DT systems. Section 3 includes background information about four main themes directly related to the project namely; smart manufacturing, the concept of digital twin technology, DT within the semiconductor industry, human computer interaction, and user experience. Section 4, designing an IDT system as a case study in the semiconductor manufacturing process. Finally, Sect. 5 concludes by reporting the progress of the project and highlights the future plans of the second and the third phases of the project.

2.1 Smart Manufacturing

Smart manufacturing (SM) is a concept which refers to the use of smart technologies within manufacturing facilities. It is seen as a newer version of what has been known as intelligent manufacturing (IM), reflecting the magnitude and impact of smart technologies such the Internet of Things, Cloud Computing, Cyber-Physical Systems and Big Data on Industry 4.0 (Yao et al. 2017). Smart manufacturing refers to using advanced data analytics to complement physical science for improving system performance and decision making (Wang et al. 2018). Smart manufacturing system include two main elements; (1) data management and (2) data analysis and these provide real-time actionable information that can be used to optimize and build system intelligence into manufacturing operations (Kuhn and Ahrens 2017). With the advances in big data and DT, smart manufacturing is becoming the focus of global manufacturing transformation and upgrading (Qi and Tao 2018).

2.2 The Concept of Digital Twin Technology

The term ‘digital twin’ is not new and it was firstly introduced by Dr. Michael Grieves, in 2003, in his Executive Course on Product Management Lifecycle (PML) at the University of Michigan in the United States (Grieves 2014). What is new; however, is the recent interest in the DT concept and applications that is fueled by the emergence of the digitization of manufacturing, cyber-physical systems, and the need to gather large amounts of data and process (big data) (Schleich et al. 2017). Moreover, the concept of Industry 4.0 has also further impacted the visibility of using more DT applications within different sectors. DT consists of three main parts: (1) physical product to be ‘virtually twinned’; (2) virtual product mimicking the physical product/twin; and (3) connected data that ties the physical and virtual products (twins) together (Tao et al. 2018). Thus, the main objective of the DT is to design, test, manufacture and use the virtual version of the physical product, process, operation, or system (Grieves and Vickers 2017).

Since its introduction, the concept of DT and its definition have been presented in several ways. For example, Boschert and Rosen (2016, p. 59) state that “the vision of the DT itself refers to a comprehensive physical and functional description of a component, product or system, which includes more or less all the information which could be useful in all—the current and subsequent—lifecycle phases”. It is also defined as the creation of the virtual models for physical objects in the digital way to simulate their behaviors (Qi and Tao 2018). The DT is a hierarchical system of mathematical models, computational methods and software services, which provides near real-time synchronization between the state of the real-world process or system and its virtual copy (Borodulin et al. 2017). Furthermore, DT is able to consistently provide all subsystems with the latest state of all the required information, methods and algorithms (Brenner and Hummel 2017). In a factory, DT is about creating and maintaining a digital representation of the real world of the factory and supporting its management and reconfiguration by the means of optimization and simulation tools, which are fed with real and updated factory data (Kuts et al. 2017).

This paper provides information about the use of DT technologies within a manufacturing plant. Based on these definitions, the DT can be summarized in three main characteristics; (1) Real-time reflection; (2) Interaction and convergence; and (3) Self-evolution (Tao et al. 2018). Real-time reflection is a key feature of any DT system. The included communication middleware within the DT system should offer real-time interworking environments that provides accurate reflection of a virtual model about physical asset and time synchronization between each simulator (Yun et al. 2017). DT systems allow real-time monitoring of the status and progress of the physical ‘twins’ through real-time data collection, data integrations, and analysis (Tao and Qi 2017). On the other hand, the DT characteristic of interaction and convergence includes three main aspects; (a) interaction and convergence in the physical space; (b) interaction and convergence between historical data and real-time data; and (c) interaction and convergence physical space and virtual space (the twins) (Tao et al. 2018). Finally, the ‘self-evolution’ characteristic of DT refers to the ability to continuously update, in a real-time, the acquired data so that the virtual models of the DT system can be continuously improved by comparing the physical and virtual twin design and configurations (Tao et al. 2018; Tuegel et al. 2011).

2.3 Digital Twin and Semiconductor Industry

The digitalization of manufacturing fuels the application of sophisticated virtual product models, which are referred to as DTs and this applies throughout all stages of product realization (Schleich et al. 2017). Since the concept of DT was proposed, it has been applied in many industrial fields and has demonstrated its great potential. DT, with the characteristics of ultra-high synchronization and fidelity, convergence between physical and virtual product, etc. has high potential application in product design, product manufacturing, and product service (Tao et al. 2017). Industry 4.0 is one of the most prevalent subjects in production engineering and intelligent DT technology acts as an important element of this fourth industrial revolution. An intelligent DT for a production process enables a coupling of the production system with its digital equivalent as a base for an optimization with a minimized delay between the time of data acquisition and the creation of the DT (Uhlemann et al. 2017). Semiconductor manufacturing processes are considered as “the most capital-intensive and fully automated manufacturing systems” and usually there are a massive number of similar equipment and tools employed in one processing line (Khakifirooz et al. 2018). Thus, it is crucial to design an intelligent DT system that manages the ingested Big Data of those manufacturing processes and leverages the interconnectivity of the machines in order to reach the goal of intelligent, resilient and self-adaptable machines (Lee et al. 2015). Based on that, DT interacts with these physical entities in the semiconductor factory together with their functions (Balta et al. 2018).

2.4 Human Computer Interaction in DT Systems

Interaction in virtual environments is different from the interaction with traditional computer-generated virtual interfaces. In the DT system there are several technologies that can be used to design accurate and nearly realistic environments for example; Virtual reality, Augmented Reality or 3D CAD simulations. In some use cases, users of such designed environments need to rotate, roll, twist and turn. Thus, this requires access to 6D elements (up, down, front, back, left, and right) in the developed systems. (Dix 2009). Human interaction has some factors that might affect how a human’s emotion and personality affect human-technology interaction (Szalma 2014). Since computers have been invented, they have become a key element in our lives and the method users interact with computers have been improved and become such an important matter. Most of the current used interactions techniques are built on some PC standard interactions tools and methods, for example, VR technology enables some new interaction technologies that use PC’s tools such as a mouse to provide some physical and natural interactions (Vélaz et al. 2014). In addition, VR has enabled some other recent interaction including gesturing, gazing, and voice interactions. Using such interactions will enhance the user’s experience when the try DT experience. The ability to design a high level of visualization and allow users to use their hands and move freely to interact with the 6D environment can provide “attractive and natural alternative to these cumbersome interface devices for human-computer interaction” (Rautaray and Agrawal 2015, p. 2). Furthermore, using eye-gazing can provide more effective interaction. In VR, the technique of eye tracking and movement has been used to calculate the user’s intention. Such interaction can be implemented to “focus depth implementation in a standard VR headset” (Pai et al. 2016, p. 171). In another recent project, it has been used to record users voice orders in real time to interact with VR environments (Kefi et al. 2018). Voice-recognition ordering can be used to interact with some objects in computer-generated 6D environments where users pronouncing some keywords representing instructions for different functions such as: pick, drop, next, and play. In this designed IDT system, voice interaction plays a crucial role and it has provided a very positive influence on the users’ experience and interaction.

2.5 User Experience

Virtual Reality (VR) technology always gave high importance to user experience (Ux) when developing interactive applications (Rebelo et al. 2012). (Olsson et al. 2013) stated the UX is related to the interaction between users and products. There are several aspects rather than the usability of products that UX assessed, such as; efficiency, effectiveness, and satisfactory interaction. It can provide more emotional relationships between the user and the designed applications (Desmet et al. 2007). Thus, UX can be defined as “a holistic concept describing the subjective experience resulting from the interaction with a technological product or service. Both instrumental (e.g. utility, usability, and other pragmatic elements) and non-instrumental (e.g. pleasure, appeal, aesthetics, and other hedonic elements)” (Olsson and Salo 2011, p. 76). Most of the human responses such as beliefs, perceptions, emotions, psychological responses, and preferences can be included in UX and can be measured and accomplished during or after using any designed system (Ritsos et al. 2011).

In virtual environments, UX was always one of the crucial elements that needed to be considered. Thus, there was a necessity to design very advanced virtual environments and applications to make users feel influenced and immersed to interact easily with the designed environmental elements. In this project, UX involves the interaction between users and the designed virtual environment application with a focus on the interaction effects between the users and the digital replica of the physical semiconductor plant.

DT has brought a lot of potential promises to overcome monitoring and visualization challenges. It made it easier to detect and predict such hidden challenges more than ever. This project contains three main levels; Top Management level, Middle Management level and Executive level. This paper focuses on the human interaction of the Top Management level. The designed DT aims to help the company’s CEOs to better understand their production lines. Thus, they become closer to the physical production plants without being personally at any production lines when they need to make any critical decisions. Since there is the need to develop a smooth interaction between the CEOs and the designed DT system, the following elements were added including 3D equipment, tools models, virtual assistant model and interactive 3D dashboard.

3.1 The Manufacturing Plant Environment

In order to achieve a high level of immersion in the IDT system, a number of natural effects were used. There is a need to improve the level of visualization in order to provide more details about all the production line’s elements. The semiconductor manufacturing plant is made of a large number of equipment, different production processes, different manufacturing levels, sections and a variety of other devices. A 3D computer aided design (CAD) design for the equipment was developed using Adobe SkeckUP, where a number of parts and component were designed to represent most of the manufacturing processes. The main reason to use CAD was to make sure to end up with 3D models that are made based on a variety of accurate and detailed components. Such methodology can be very helpful to design complex assets and help the DT systems to provide more information about those assets. Figure 2 shows a screen shoot for the designed DT of the manufacturing plant environment in use in our test system.

(Source: Self).

A screen shot of the virtual wire bonding production line

Full size image

In addition, this approach will allow users to interact directly with the different components and better understand their behaviour. Gaze detection together with VR controllers will help users focus on any part of the machines and visualise the require statistics when they are needed. Users will be able to walk and pick those virtual parts using a leap motion device a computer hardware sensor that supports hand and finger motions as input. Such interaction can be enhanced with voice recognition in order to elicit further explanations when interacting with the developed virtual assistant. Figure 3 shows a diagram that explains one of our scenarios whereby we can see the interaction between the users and the virtual 3D equipment.

(Source: Self).

The gesturing interaction diagram

Full size image

3.2 The Virtual Assistant

From the early stages of development on this project, a decision was made to give a greater attention to the level of interaction between users and the designed system. In recent years, a good number of new interaction techniques have been developed. One of these recent methods was used to provide a user-friendly interface that allows users to interact smoothly with the digital replica of a very complex semiconductor manufacturing process. The system is built on a live stream of data that was gathered in a real-time manner from the manufacturing line. As a matter of fact, semiconductor manufacturing provides a massive amount of data about all the different manufacturing processes, materials, and operators. After evaluating monitoring system of an international semiconductor manufacturing company that we conduct research with, we have realized that is very difficult to control the fast flow of data through interactive dashboards. It has always been a challenge to determine which parameters should be given priority in order to make fast and accurate decisions. In addition, using traditional interaction techniques such as; mouse, keyboard, and touch devices, can cause delay and damage when dealing with emergency situations that require instant attention.

An Artificial Intelligence virtual agent was designed to take care of all the users’ required information. Users can interact directly via speech orders to ingest information about the manufacturing processes through the virtual DT system. In addition, and to add more realistic interactions through the digital replica, user’s movement capability was added to allow users to move from equipment to another while they are investigating unforeseen failures that might occur in the manufactory process. Natural language processing using machine intelligence was used to translate users’ voice commands into written words. IBM Watson Speech-to-Text API^{Footnote 1} with unity IBM-Watson-SDK was used to read voice commands which are used to invoke other functions such as; “move to the failure source” or “show statistics that are counted form the collected data in real-time”. Figure 4 illustrates the applied voice interaction.

(Source: Self).

Voice interaction diagram

Full size image

This interaction has a number of advantages; for instance, the used hardware, can use this easy-to-use service that uses machine intelligence to apply language grammar and generate very accurate transcription. Also, the API supports 7 different languages which can make it easy to gather the feedback from the virtual assistant and translate it into different languages. This feature will make it easy to distribute the designed model over different plants and can be used by different users without the need to make major changes in the language packages used. In addition, the approach used does not require expensive microphone hardware since it correct the voice and grammar mistakes to make sure the virtual assistance gets accurate and correct orders. Another IBM Watson API was used, the Text-To-Speech API. This toolkit was used to invoke different functions based on the user’s original voice order and to provide the requested information in very speedy way thus avoiding issues of. Figure 5 shows the methodology for the virtual assistant response.

(Source: Self)

Voice recognition interaction with the virtual assistant

Full size image

3.3 3D Statistics Visualization

To achieve this, all the decision makers need to have enough information to make accurate decisions. During this project, a new approach was designed to make the user’s interaction easier by visualising all of the collected statistics pertaining to the manufacturing process in real-time. Both 3D and 2D diagrams were added to the DT system to be available whenever they are needed. Two main interaction tools are used including VR Controllers and the leap motion to help users select, remove, pause and refresh all the designed systems. The DT environment includes a virtual section for data analysis. As it can be seen in Fig. 6, this section contains a number of screens that can be interacted with. In the Cave VR experience, the gestures captured through the leap motion are used to interact with the 3D diagrams. Apart from the VR controllers and leap motion tools, the developed virtual assistant can help users to answer any questions when they are need. Even though the project is still under development, the initial feedback from the top managers was very positive.

(Source: Self)

A screen shot of the designed virtual dashboard

Full size image

3.4 Image and Sound Effects

Since some of the DT virtual equipment are mechanical components, there was a need to implement different techniques in order to improve the user’s interaction when there is a sudden or predicted failure. In order to boost user immersion, a number of sound elements were added. Using 3D binaural sounds which form part of the Unity Library, we can help to signal any unusual behaviour of any equipment in the manufacturing plant. At this stage, only normal equipment’s sounds were used. Examples of the used sounds were; fans, motors, solder bonding headers and tracks. The current data ingested does not include any parameters which can be used as part of the designed DT model. On the other hand, a number of imaging effects such as Bloom, Depth of Field, Tone mapping or Colour Correction were implemented to mimic different equipment status in case of drop-down events. Such effects can definitely help to catch the users’ attention in order to respond to failure events. Figure 7 shows an over heat scenario that was visualized using image effects. Using 3D binaural sounds, users can respond to sound sources and grab their attention so that they only focus on the source that has problems.

(Source: Self)

Visualized over-heating scenario in the designed DT system

Full size image

4.1 Results

The use of DT in manufacturing has a promising future. After a number of months working on developing this project, there were a number of key achievements that have been accomplished. A crucial key element that has added more value to the developed system was the user interaction experience. By implementing a number of different human interaction techniques, the system has become more user-friendly and interactive and thus helped users to make faster reactions. The designed virtual assistant has successfully improved the interaction between the users and the designed DT system. There is also the possibility to enhance it further by providing more answers and suggest more explanations to the users. Virtual assistant makes the user experience even easier and helps users to interact with the whole system thus reducing the dependency on the virtual hardware and the visualized statistics.

In the coming years, it is expected that an enriched big data environment incorporating smart manufacturing ideas such as intelligent DT will further enable advanced analytics. As a result, visual computing will become a new and exciting field of research linked to the challenges of the next industrial revolution. Thus, it is crucial to design an intelligent DT system that manages Big Data and leverages the interconnectivity of machines in order to reach the goal of intelligent, resilient and self-adaptable machines besides the high capability of improving the user experience and interaction within the DT system. There were obviously a number of which provided useful insights into the applicability of improving user interaction and UX using the intelligent DT technology within the manufacturing industry. Such insights will be beneficial to future human-computer interaction approaches within DT technology. Realistic features can be added, smoother interaction can be implemented, and big achievements can be obtained as a result of combining these advances.

4.2 Future Work

The project is at the implementation and development for the first phase. Based on the promising results that the project has gained so far, more attention will be given to the Conversational Virtual Agent via different forms of interaction with agents, including chat, voice and 3D interaction. In addition, the leap motion interaction will be developed in the future to help users interact with a virtual laboratory (VL). In this VL, the engineers will be able to interact with virtual products and test different scenarios virtually. It will help engineers to test virtual products using real-time data and historical information from the previous scenarios in the DT system. Thus, there will be no need to interrupt any physical production processes to test new improvements. Based on the initial feedback and results obtained so far from this project’s work, we will be implementing more interaction techniques such as; auto gazing to pick up and select equipment for the big data system. The VR technology will be used to design a DT system that can be displayed using a head mounted display. This AR aims to provide a window over the DT on handy tablets in order to allow engineers to work freely through production lines, simulate, discuss and suggest future procedures in real time.

Digital twin technology is capable of integrating various technologies in one robust system. This includes the capability of implementing and improving users’ interaction techniques within DT systems. Such interactions can reflect on the user’s experience while trying the designed DT. This project aimed to design a realistic DT system to immerse users in digital replica of physical assets in an attempt to make decision makers become closer to their manufacturing processes. This project had a number of key achievements and integrated virtual reality interactions together with leap motion interaction to help users better understand the manufacturing process and smoothly interacting with all the digital twin’s assets. The initial results have shown a great potential in the proposed interaction techniques and managed to design a user-friendly experience through the use of the designed DT system.