[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Next Article in Journal
Robust Handover Optimization Technique with Fuzzy Logic Controller for Beyond 5G Mobile Networks
Previous Article in Journal
A Model between Near-Surface Air Temperature Change and Dynamic Influencing Factors in the Eastern Tibetan Plateau, China
You seem to have javascript disabled. Please note that many of the page functionalities won't work as expected without javascript enabled.
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sensors
Volume 22
Issue 16
10.3390/s22166201
sensors-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection

by
Oleg O. Kartashov
*,
Andrey V. Chernov
,
Alexander A. Alexandrov
,
Dmitry S. Polyanichenko
,
Vladislav S. Ierusalimov
,
Semyon A. Petrov
and
Maria A. Butakova
The Smart Materials Research Institute, Southern Federal University, 178/24 Sladkova, 344090 Rostov-on-Don, Russia
*
Author to whom correspondence should be addressed.
Sensors 2022, 22(16), 6201; https://doi.org/10.3390/s22166201
Submission received: 25 July 2022 / Revised: 12 August 2022 / Accepted: 16 August 2022 / Published: 18 August 2022
(This article belongs to the Section Fault Diagnosis & Sensors)
Browse Figures
Figure 1
<p>Connection diagram for Raspberry Pi4 modules and expansion boards.</p> "> Figure 2
<p>General network architecture of used YOLOv5.</p> "> Figure 3
<p>General view of the process of restoring the depth of an object based on a binocular approach.</p> "> Figure 4
<p>Results of preprocessing images of welded joints radiographic inspection.</p> "> Figure 5
<p>Samples of marked welds in steel pipelines images.</p> "> Figure 6
<p>Stages of preliminary processing of image data.</p> "> Figure 7
<p>Examples of preprocessed and labeled image data.</p> "> Figure 8
<p>First case model training and validation metrics for a set of counter images.</p> "> Figure 9
<p>Mutual dependencies of the model metric indicators used (<b>a</b>) precision–confidence, (<b>b</b>) precision–recall, (<b>c</b>) recall–confidence, and (<b>d</b>) F1–confidence.</p> "> Figure 10
<p>Second case model training and validation metrics for a set of counter images. (<b>a</b>) Correlogram of ROI labels; (<b>b</b>) Distribution density of image labeling parameters; (<b>c</b>) Confusion matrix; (<b>d</b>) Standard set YOLOv5 training and validation metrics.</p> "> Figure 11
<p>Mutual dependencies of the model metric indicators used in the second case: (<b>a</b>) precision–recall, (<b>b</b>) recall–confidence, (<b>c</b>) precision–confidence, and (<b>d</b>) F1–confidence.</p> "> Figure 12
<p>(<b>a</b>) Average ROC (<b>b</b>) ROC area to multilabel classification, (<b>c</b>) PRA map showing dependencies average precision, recall, and F1-score metrics. (<b>d</b>) Confusion matrix.</p> ">
Versions Notes

Abstract

:
During the steel pipeline installation, special attention is paid to the butt weld control performed by fusion welding. The operation of the currently popular automated X-ray and ultrasonic testing complexes is associated with high resource and monetary costs. In this regard, this work is devoted to the development of alternative and cost-effective means of preliminary quality control of the work performed based on the visual testing method. To achieve this goal, a hardware platform based on a single board Raspberry Pi4 minicomputer and a set of available modules and expansion cards is proposed, and software whose main functionality is implemented based on the systemic application of computer vision algorithms and machine learning methods. The YOLOv5 object detection algorithm and the random forest machine learning model were used as a defect detection and classification system. The mean average precision (mAP) of the trained YOLOv5 algorithm based on extracted weld contours is 86.9%. A copy of YOLOv5 trained on the images of control objects showed a mAP result of 96.8%. Random forest identifying of the defect precursor based on the point clouds of the weld surface achieved a mAP of 87.5%.

1. Introduction

Pipelines are essential strategic transport arteries for all countries around the world and play an important socioeconomic role for large energy and chemical companies. Within the framework of small subjects of states, pipelines serve as an integral tool for delivering critical natural resources and coolant to the final consumer: citizens. This type of transport has practical advantages, including environmental friendliness, the ability to operate under adverse weather conditions, the continuity of the resource delivery process, and optimization of the workload of the remaining modes of transport [1]. Pipelines are widely used today for the transmission and distribution of the most popular energy resources, such as gas and oil products, and for modern, promising, and more environmentally friendly initiatives, such as coal water slurry [1,2,3]. Restrictions imposed on the use and development of transport pipeline networks in some countries can significantly reduce general trends in developing the country’s technological and socio-economic potential [4]. Important aspects of the construction of such networks are the planning of their placement, the settlement of legal aspects of the construction of engineering structures, the quality control of the pipeline installation, and the compliance with all related bylaws and standards [1,5].
The primary materials for the installation of pipeline lines are steel, plastic, and propylene. Although each option has its advantages and disadvantages [6,7], the majority of popular choices for most of the networks currently under construction are steel. One of the most vulnerable places to install steel pipelines is the joint surface, which is made using welding technologies. Nondestructive testing methods (NDT) [8] were recently widely used to perform visual quality control checks of the surface of welded seams. The main nondestructive testing methods of the pipeline welded surface according to Table 341.3.2 of ASME B31.3-2004 [9] are visual inspection, liquid penetration inspection, magnetic particle testing, radiographic inspection, ultrasound testing, and eddy current testing. According to the data presented, the most effective method to recognize most weld imperfections is the radiographic method; however, visual evaluation allows you to identify approximately 70% of the types of defects. Based on the international standards ISO 5817:2014, ISO 6520-2:2013, and ISO/FDIS 9712-1999 in fusion welding parts, we can extract the key features of steel pipeline weld surface inspection and highlight the required list of defect detection accuracy.
There are many software and hardware systems and technologies for implementing methods for the nondestructive testing of girth welds in pipelines [10,11] that meet the requirements of the standards listed above. Radiographic and ultrasonic examination methods gained the greatest popularity [8,12]. However, both approaches are quite resource intensive. The high cost of equipment or consumables, significant time costs, and personnel qualifications are key factors that slow the pipeline construction process by contractors performing installation work. During pipeline line commissioning, licensed and accredited NDT laboratories are involved, which significantly guarantees the safety of the installations. In this case, the use of expensive equipment is fully justified. However, conventional contractors often require an affordable and sufficiently accurate means of assessing possible weld defects for a preliminary assessment of the quality of the steel pipeline being laid. Another important aspect of implementing control methods is their automation, which will significantly reduce resource costs during the construction process and ensure the optimization of the staff during line installation. Since the use of ready-made defect templates, industry standards for the enterprise do not provide high accuracy in defect assessment. The participation of an expert in the interpretation of sound spectra or X-ray images requires considerable time, considering the human factor, which does not always have a positive effect on the construction process: new methods are needed for decoding and classification of defects of the control object. Researchers are increasingly paying attention to the use of artificial intelligence technologies, which, together with sources of sensory perception of information about the progress of the welding process, allow the implementation of highly efficient algorithms for determining defects in the test object. First, we will highlight several stages to conduct circumferential weld inspections. A preliminary assessment is carried out directly during the construction process to obtain the minimum necessary to confirm information for the continuation of work. The next stage is the primary check, carried out before commissioning of the pipeline transport network. The final stage is a set of measures for the maintenance and inspection of pipelines in operation, among which are the methods of in-line inspection, such as magnetic flux leakage control (MFL), ultrasonic testing (UT), electromagnetic acoustic technology (EMAT), and eddy current testing (EC) [13].
In the further article text, an existing solutions mini-review will be presented, materials and methods will be identified, data preparation issues will be considered, the study main results will be analyzed, discussion will be held, and a conclusion will be made.

2. Existing Solutions Mini-Review

In this paper, we focus on the preliminary assessment stage of the visual control of the surface of steel pipeline girth welds. Thus, we considered several studies that allow us to determine the current state of the problem with respect to the implementation of determining defects regarding the most popular methods of NDT. For example, the paper [14], due to a detailed review of existing technological solutions, demonstrates the applicability of individual artificial intelligence methods and their advantages and disadvantages in organizing a defect classification system during the ultrasonic and radiographic studies of test objects. There are approaches to the analysis of radiographic images in which the authors proposed a hybrid approach that combines deep learning (DL) models based on the use of the AlexNet network and autoencoders. For example, these are artificial immune systems (AIS) [15]. According to the information provided by the authors, this approach makes it possible to significantly improve the accuracy of determining weld defects relative to the separate use of the indicated methods. The study [16] involves the initial extraction of the weld area from the radiographic image, which is analyzed using the traditional approach of extracting signs of a defect, such as threshold segmentation. In the future, it is planned to use the support vector machine (SVM) to detect blocks with defects. As a result, the authors determined the high classification ability of SVM, but noted the sensitivity of the results of applying this method relative to the original training datasets. Some research is focused on a more comprehensive approach to solving the problem, so in [17], a detailed description of the method is given to study weld defects using layered ultrasound arrays. The authors fully worked out the theoretical and practical aspects of the application of this method, including industrial use. The prospects for the use of automated ultrasonic testing (AUT) are outlined, and the results of multivariate analysis using principal component analysis (PCA) are presented. Some researchers are already concentrating their efforts on the development of existing methods and results. Thus, in [18], the authors define the procedure for X-ray image verification by qualified specialists and the need to automate this procedure. The study above used DL and transfer learning (TL) technologies based on pretrained deep convolution neural network models and a comparison of deep convolutional activation functions (DCFA). There are approaches to detect defects in steel pipeline welds based on the use of computer vision and machine learning algorithms. For example, in [19], the authors used the YOLOv5 algorithm and compared it with the two-stage algorithm to detect objects, Faster R-CNN. It should also be noted that the imaging setup in this study was a real-time X-ray imaging system consisting of a movement mechanism for the test tube, an X-ray machine HS-XY-225, a high-speed digital detector PS1313DX, and an image capture card [19]. In [20], attention was mainly paid to the detection of defects in the welds based on the localization of the defect by applying morphological filtering, further specifying the type of defect using a convolutional neural network (CNN) and the defect zone using SVM. Separate lines of research include the analysis of magnetic flux leakage imaging (MFL) images, where a gradient coincidence matrix (GGCM) was proposed as a defect recognition tool [21]. In some cases, to extract information on the presence of a defect from radiographic images, the Faster R-CNN algorithm is used, where the authors identified two extractors of internal features and evaluated their effectiveness in [22]. In general, the use of various types of images and spectra of sound sequences is one of the most sought-after areas of research in this area. Thus, the paper [23] reviews possible approaches to the analysis of X-ray and infra-red images of test objects and methods to determine defects based on the use of acoustic emission sensors and ultrasound. Although the study [24] is more focused on monitoring and analyzing the quality of the welds during pipeline operation, it allows you to get a description of such inspection methods, as the use of a three-axis high-definition MFL detector, an in-line ultrasonic crack detector, and the structure of an embedded detector with EMAT. In addition, we disclose the applications of technologies for electromagnetic eddy current testing and the verification of field measurements of alternating current (ACFM). The studies carried out in work [25] confirm the high efficiency of using the radiographic verification method under various conditions and allow the extraction of comprehensive information on welding equipment, the welding process, and the acceptable deviations in the evaluation of defects. Automatic systems that classify defects in steel pipe girth welds based on radiographic images using various machine learning models are the main proposal for the implementation of quality control verification in the installation work performed by welders. Thus, in [26], it is proposed to use the KNN and SVM methods to determine linear defects in pipeline welds. Such an integrated approach, according to the results obtained, guarantees an increase in the classification accuracy. In the study [27], the TL technology was implemented in conjunction with CNN. Here, traditional image processing techniques (IPT) were used to preprocess radiographic images. Descriptors extracted from the GDXray database using VGG16 and ResNet50 were used as data in the training sets of classifiers. The authors used random forest, support vector machine, and logistic regression as classification models. Regarding the results obtained, the authors present data on the increase in productivity and the high accuracy of these methods. Some studies offer alternative approaches to the calculation of weld defects, for example, the use of coarse set theory to create a classification system [28] or the implementation of linear control methods based on data management [13].
Based on the above information, we can conclude that the main methods of the nondestructive testing of girth welds of steel pipelines today are radiographic and ultrasonic testing methods. Furthermore, the wide possibilities of artificial intelligence methods offer an extensive toolkit for automating the identification and classification of possible defects in control objects. However, the purpose of this work is to create a simple and economical means of preliminary assessment of the quality of the installation work. In this study, a control system based on a visual inspection method is proposed, which in any case will not be able to recognize all types of weld imperfections. However, this tool will be able to detect in advance about 70% of violations of the weld manufacturing process, including cracks, lack of fusion, incomplete penetration, surface porosity or inclusion of exposed slag, surface finish, concave root surface, weld reinforcement, or internal protrusion, which in most cases guarantees increasing the efficiency and quality of work by contractors performing the installation of pipelines.

3. Materials and Methods

We chose a Raspberry Pi4 single-board computer as the main node to collect, broadcast, and preprocess visual information to develop the proposed software and hardware tool to inspect pipeline weld surfaces. This choice was based on the availability of the board, its low power consumption, extensive plug-in and expansion card capabilities, the presence of a built-in wireless module, and adequate performance for computing procedures and imaging research experience [29,30,31,32,33]. Since our project involves the connection of two camera modules to be able to build a depth map of the surface of the control object and an inertial measurement unit to determine the location and spatial orientation of the video information acquisition device, there was a problem with the presence of only one scalable coherent interconnect (SCI) port.
This problem is proposed to be solved using a perfectly suitable IVPort V2 Raspberry Pi Camera Module V2 multiplexer [34]. This option allows you to connect to four camera modules simultaneously, which is essentially a hub from one SCI port to four. The main disadvantage of this board is the decrease in framerate for each of the connected cameras, since data capture is available from only one source connected through this board at a time. Since this board also has limitations on possible camera plug-ins, we chose the Raspberry Pi Camera Module V2 with 8MP SONY IMX219 sensor. The equivalent focal length of 33 mm, a sufficient lens aperture, and a wide range of supported video formats make it possible to use these modules in our project. The spatial orientation of the device is determined and assumed using the Kootek GY-521 MPU-6050 IMU modules for Raspberry Pi that operate on the IIC protocol with a built-in 16-bit AD converter. As a means of providing location data, the equipment is planned to use an L80-39 GPS module with a L80-39-based USB interface. The hardware platform is powered by another Raspi UPS HAT lithium battery expansion board, with 3.7 V specifications and a capacity of 2600 mAh. This reserve will allow the hardware of the proposed tool to operate for about two hours. The equipment connection diagram is shown in Figure 1. The circuit development was carried out with a cross-platform software package of the electronic design automation (EDA) class. It was also used to develop circuit diagrams and layouts of printed circuit boards.
Since it is supposed to study the girth welds on different diameters of pipes, the mechanical parts of the hardware platform suggest the possibility of adjusting the relative position of the cameras close to the object. The placement of onboard equipment requires the presence of a trolley with the possibility of uniform movement along the surface of the pipe. Transmission of preprocessed data to the user’s workstation is carried out by organizing the direct connection mode (ad hoc).
The methods implemented that are used in the development of the software part of the tool for diagnosing pipeline welds can be conditionally divided into several stages. The initial stage of the operation of the diagnostic tool is devoted to the collection of several data streams, including geolocation (data from the IMU and GPS module) and video data streams. Using the data received from the GPS module, a geotag is formed about the position of the pipeline joint diagnostics. This information will be used to generate a unique ID for each of the measurements. In addition, in the future, it will be possible to trace the control checks carried out on a digital map. Since, in our case, there are limitations to the possibility of synchronous video capture, the parameters of the initial video stream will be a resolution of 640 by 480 pixels at a frame rate of 15 frames per second. Since each module will perform the recording in turn, it is required to compare the frames of the video recording made by one module relative to another. When switching video recording channels, the delay is up to 1 s, which in some ways imposes restrictions on the speed of movement of the inspection tool trolley along the pipe surface. In our case, the IMU is needed to find the central shooting positions of the girth weld, which are sequentially different from each other by 45°, starting the report relative to the vertical axis of symmetry of the pipe. Time ranges are required to converge IMU readings with intermediate video recording measurement values, where the data stream from the measurement IMU is matched with the timeline of the measurement being performed, and then the required time values are extracted at the measurement control points. The time values obtained are then used to select and extract the appropriate keyframes from the video data streams, assigning a measurement position number. Direct extraction of frames from a video is performed and they are saved in an image file; then, a set of FFmpeg libraries from the Python procedural language is used. The effect of the impact of the encoder used in this set of libraries was determined in [35]. The name of the image file is derived from the number of camera modules, the measurement location, and the coordinates at a given time. Thus, in the output of this stage, we received two sets of image data, eight attachments each, where the file name contains additional information necessary for further use. The next step is to pre-process the image data files themselves. This section contains introductory material on the data preprocessing procedures. We will look at this in more detail in Data Preparation. In our case, the resulting visual data sets will be used in the future to solve two problems, the first is to identify and classify defects in the pipeline weld, the second is to make a three-dimensional reconstruction of the surface of the object under study with a color indication of areas with imperfections. Consequently, for each of the tasks to be solved, the preprocessing methods will differ. Since the measurement process does not take place under ideal conditions of illumination of the surface of the test object and there is a presence of extraneous noise in the image, it is proposed to use the median filter algorithm. The application of this approach can significantly improve the visual representation of the image while eliminating a significant amount of noise and equalizing the brightness of each image pixel [36].
The software implementation of the procedure for restoring the images obtained during the collection of information was performed using the OpenCV library in the Python programming language. Next, to implement the system used to classify weld defects, the resolution of the images in the studied sets was reduced. When designing a system to detect and classify defects in steel pipeline welds based on the nondestructive visual testing method, we noticed that most of the deviations depended primarily on the morphological properties of the object under study in the image. Therefore, an integrated approach to solving the problem was considered. Morphological characteristics were best demonstrated first by images of radiographic inspection of welded joints. In our case, the attribute space of such data sets is redundant, since, in addition to surface defects, they describe internal imperfections of welded joints. However, we discovered the possibility of extrapolating the results of the intellectual analysis of sets of radiographic images for the reliable identification of weld defects based on the visual method. Data sources for the study were the public radiographic image dataset GDXRay+ [37] and the radiographic and visual image datasets provided by LLC «Cyber Vision». YOLOv5 [38] was chosen as an algorithm to identify weld defects in images due to mosaic processing of graphic data, which allows qualitative extraction of features characterizing small rigid objects. The general network architecture used by YOLOv5 is shown in Figure 2.
The main large parts of YOLOv5 are backbone, neck, and head. The first one extracts and forms image features with different detail levels. The next one is used for feature aggregation. The last makes predictions based on features from the neck. For backbone used Cross Stage Partial DarkNet53 (CSP DarkNet) with spatial pyramid pooling. This serves to increase the receptive field at separating features without compromising performance. For the neck, the PANet solution is used, which provides an adaptive pool of features connecting the mesh with all levels of features, for uniform propagation in the following steps. The Yolo layer in the head implements dense prediction for a one-stage detector type. In general, the choice in favor of YOLOv5 was also made on a few considerations. Thanks to CSP DarkNet, this algorithm has high speed and accuracy of inference. PANet allows for more efficient low-level feature propagation, and the dense prediction layer makes it possible to implement a multiscale prediction, which is quite useful for the condition of various morphological characteristics of weld defects. In addition, in the study [19], the authors carry out comparative tests on a number of approaches to one-stage and two-stage detection, where YOLOv5 showed excellent results. It is especially worth noting the superiority over Faster R-CNN in terms of speed, accuracy, and final weight of the model. Based on the data used in [19], we considered it possible to extend their experience to our study. The ability to detect and classify in real time, the low weight of the trained model, the speed and issuing efficiency of the predictions, the simplicity of data labeling, and training make YOLOv5 an advantageous candidate for solving our problem. This algorithm was trained twice. In the first case, a Gaussian blur with a radius of 5 was applied to the sets of radiographic images, and the edges of the objects were extracted using the Prewitt operator. Additionally, we made changes to the data labeling by eliminating hidden defects that are inaccessible to the visual method of control. The resulting modified data sets were used to train YOLOv5, where the contours of the test objects extracted from conventional weld images using the same Prewitt operator were used as a test set. In the second case, data characterizing surface defects were selected from the sets of radiographic images and the model of the YOLOv5 algorithm was trained, which was further trained on the labeled data of ordinary images. This was done on the basis that there were about 8000 examples of radiographic images, divided into 6 classes of welding defects, and there were about 1500 ordinary images, which could not guarantee a confident prediction of the presence of a defect in itself. In the process of shooting controlled seams with two cameras, we restored the depth of objects; a general view of this procedure is shown in Figure 3.
To reconstruct the depth map, algorithms for searching for local features in images were used, since the algorithms for searching for global matches are more computationally expensive, and this contradicts the proposed concept of the approach. The depth maps obtained were converted to point clouds on the surface of the welded joints of steel pipelines. The extracted point cloud data were used to detect and classify weld defects at the level of their morphological characteristics. For this, a modified method was used with further analysis based on the random forest model [39,40,41]. We implemented and developed the procedure for the three-dimensional reconstruction of the surface of the pipeline weld.
Because of the shooting of images is performed in key positions, there are overlap zones (repeating visual information in two images) and distortions of the weld geometry relative to the shape of the object under study; when merging eight images of the weld, it is necessary to refine the overlap zones and correct the shape of the weld on the image. Standard tools provided by the OpenCV library were used to implement the image splicing procedure in software. After obtaining two spliced images relative to visual data sets from two multidirectional camera modules, it is required to build disparity and depth maps. Next, we use the software implementation of the computer vision algorithm, efficient large-scale stereo matching [42]. Based on the results of third-party research conducted in other areas [43], quite good results are guaranteed in our case, with relatively low complexity of implementation. After obtaining disparity and depth maps, it becomes possible to build an red, green, blue, depth matrix (RGBD) for the image of the weld. Then, using geometric arguments, we implement the process of converting the screen coordinate system to a Cartesian coordinate system. This idea was born from the reverse engineering results presented in [44]. However, despite obtaining a point cloud of the weld surface, it is worth recalling the geometry of the test object, the pipe, of which, as a result, the three-dimensional reconstruction should be a three-dimensional ring. We used the values of the radius of the curvature of the pipe relative to the size of its diameter as a correction factor that increases with each step for the location of each cloud point in the Cartesian coordinate system. Then, the reconstruction of the surface mesh [45], with respect to the coordinates of the point cloud, is assumed. The software implementation of this procedure was performed through a convolutional neural network with an integrated voxelization graph (LV-GCNN) [46], which allows reconstructing the surface without visible losses. The resulting digital model is then divided into sections according to the initial shooting positions.

4. Data Preparation

In this study, a total of four initial datasets were used. This is a public GDXRay data set, or rather, a part of it containing X-ray images of welded joints. A set of weld radiographic images (8000 images) and a data set of weld photographs (1500 images) were provided by Cyber Vision LLC. In addition, we collected a dataset of point clouds of steel pipeline weld surfaces (100 surfaces). In addition, on the basis of existing sets of graphical data, a set was created that contains images of the extracted contours of the welds. In the process, all datasets were labeled or relabeled and pre-processed.
We selected two data preprocessing scenarios based on the problems of detection and classification of defects and the three-dimensional reconstruction of the surface of the test object. As part of the first procedure implementation, images of two types were used: the usual visual and radiographic representations of the test object. The radiographic images were subjected to a Gaussian blur procedure with a 5 radius and further extraction of the edges of the objects in the image using the Prewitt operator. The results of this processing are shown in Figure 4.
For processing graphic materials of visual data sets of weld images, the methods used in our case can be divided into setting the basic resolution, filtering (noise removal and brightness equalization), segmentation, and color space transformation. A resolution of 224 × 224 pixels was defined as the reference resolution for sample images. For this, software procedures for cropping frames and reducing their resolution were implemented. At this point, image data pre-processing procedures were suspended in favor of markup. The marking of the images was completed manually by an expert experienced in the visual assessment of weld seams. The procedure for labeling images of welds was carried out using the LabelImg software, which has a clear graphical interface and wide functionality for marking data sets. In the process of marking these images of welded seams of steel pipelines, we were guided by two international standards, in particular ISO 5817-2021 and ISO 6520-2-2021.
Based on the standards considered, six main types of welding defects and their descriptors were identified for the implementation of the visual method of nondestructive testing. The identified types of defects include: (1) cracks; (2) cavities; (3) solid inclusions; (4) non-fusion; (5) form defects; and (6) other defects. Examples of marking images of welded seams of steel pipelines are shown in Figure 5.
At this stage, the first set of visual data was formed, characterizing the defect area of the test object based on the color space of RGB images, which is useful for identifying such defects as, for example, inclusions.
For better detection of other types of defects, it is necessary to highlight the most informative features of visual data. For this, three stages of preprocessing graphic data are proposed. All procedures for pre-processing image data are shown in Figure 6.
In the first stage, the image color space was converted to grayscale. This procedure was implemented using the multiparadigm Python programming language and the scikit-image library. The next step was dedicated to segmenting the image to highlight the contours of the weld against the background of the steel pipe. For this, a spatial filtering method based on a sliding mask with a dimension of 3 × 3 pixels was applied, and a Gaussian blur with an intensity of 3 was previously used to eliminate excess noise. The coefficients of the mask square matrix were determined using the Prewitt operator. After successful selection of the weld contour, blurring of the image background was applied using the Gaussian function and an intensity of 10. Examples of the images obtained are shown in Figure 7.
The second scenario does not include reducing the resolution of the original image, but implements the procedure for merging all images within each of the two sets. The Random Sample Consensus (RANSAC) and Scale-Invariant Feature Transform (SIFT) algorithms were used for image splicing, which reveal local features with which you can check the similarity of the original images in the set. In addition, we used the SIFT algorithm to calculate and represent the three-dimensional surface points of the testing object based on two spliced images. The pseudocode for the software implementation of this function is given in Algorithm 1.
Algorithm 1. Calculation and presentation of points of a three-dimensional controlled object/procedure CalculatePointCloud(imgL, imgR).
INPUT: Two images of calibrated cameras
OUTPUT: Point cloud
        imgL:= ConvertColor(imgL, GRAY)
        imgR:= ConvertColor(imgR, GRAY)
        keypoints_1:= SiftDetectAndCompute(imgL)
        keypoints_2:= SiftDetectAndCompute(imgR)
        keypoints_1, keypoints_2:= Match(keypoints_1, keypoints_2)
        F:= FindFundamentalMat(keypoints_1, keypoints_2)
        pointCloud:= Triangulation(F, keypoints_1, keypoints_2)
        return pointCloud
end procedure
Additionally, within the study framework, an approach was proposed and implemented to assess the defectiveness of the welds based on the analysis of a cloud of points on the surface of the test object using random forest. To do this, point cloud data was collected using the Intel Realsense D435i intelligent sensor camera, followed by the extraction of the required values from the bag file. It was these data that were used later to train the random forest. The real test data sample was formed by algorithmic processing of two video data streams and the subsequent transformation of the obtained depth maps into point clouds. A fragment of the structure of the received point cloud data is shown in Table 1, and the raw data and other open results of the ongoing research are available in the public repository of projects at the link: https://github.com/cybervllc/real_soft (accessed on 22 July 2022).
Marking of these point clouds was carried out by selecting a sequence of points belonging to the defective zone of the weld or defining other morphological characteristics that do not meet the requirements of the international standards in this area. As a result, each investigated weld fragment had 11,000 to 13,000 points, and the average number of points to describe the defective zone was in the range of 1500–2000 points from the entire fragment array.

5. Results

The proposed approach to assessing the defectiveness of a weld based on processed video data streams is to implement a deliberative procedure of three trained machine learning algorithms, including YOLOv5 to identify anomalous contours of objects, YOLOv5 to detect defective zones in images of a weld, and random forest to analyze spatial data point clouds of the surface of the welded joints of steel pipelines. The deliberative procedure basis is the principle of two of three. This means that at least two of the three algorithms must detect a defect in the object under study, even if they identify a different type of deviation. The training of all test models took place on a stand that included a discrete adapter Nvidia Quadro RTX8000 with a memory capacity of 48 GB, an Intel Xeon E5-4655 v3, and 128 GB RAM with the Linux Ubuntu 18.04.6 operating system on board. To train the YOLOv5 first case, the hyperparameters set given in Table 2 was used.
The results obtained during the first computer vision algorithm training to detect defects based on the image contours of a steel pipeline weld are presented in the graphic materials below in Figure 8. Here, Figure 8a shows a correlogram of ROI labels, which is a group of two-dimensional histograms showing pairwise projections of the data axes according to the “each with each” principle. Figure 8b contains information about the distribution density of image labeling parameters relative to all available classes. The confusion matrix of first case YOLOv5 is given on Figure 8c. The upper row Figure 8d shows the model training metrics. Specifically, in order, root mean square error (RMSE), binary cross-entropy, entropy, precision, and recall. The bottom row Figure 8d shows the same set, only for model validation.
Additionally, for a qualitative result obtained from the examination, we collected pairwise graphs of the dependence of the quality metrics of the algorithm used, such as precision, recall, f1, and confidence (Figure 9).
This result allows us to assert the high accuracy and stability of the work of this algorithm. The training of the second copy of YOLOv5 took place in two stages. At the first stage, the algorithm was pretrained on radiographic control images. At the second stage, it was further trained on preprocessed and marked images of welded steel pipes. To train the YOLOv5 s case, the hyperparameter set given in Table 3 was used.
The final metrics to train the algorithm and the quality of the model are given below. The Figure 10 structure is identical to the Figure 8 structure, except that it contains an evaluation of the YOLOv5 second case.
In addition, as in the first case, graphs of mutual dependencies of the metric indicators of the model used for its characterization were prepared (Figure 11).
In addition to these algorithms for the detection and identification of defects in welded joints, an approach based on the analysis of the obtained point clouds of the object surface under study was used. Strictly speaking, we determined each point potential from the set belonging to a possible defect using the random forest algorithm. In this case, the point clouds belonging to surface defects and described above served as the initial data set. This approach concept is as follows: we consider independently each point from the points cloud corresponding to the one-object measurement and present the probability that this point belongs to one of the six defects. Next, we integrate the number of points that belong to the same defect in the areas of the surface under study, having previously clustered them. If the checksum of the number of points per potential defect exceeds the threshold value, the system determines this as a precursor of a weld defect. Since we know in advance the geometry of the control measurements and the survey scale, the values of the threshold values of the control sums were chosen on the basis of the average geometric dimensions of real defects and are unique for each type of defect. Data on the training and testing processes of the random forest algorithm are shown in Figure 12.
As a result, we prepared and trained three algorithms, YOLOv5 to detect defects based on images of weld contours with an [email protected] value of 86.9%, YOLOv5 to detect defects in welded joints based on images of the test object with an [email protected] value of 96, 8%, and random forest to detect the belonging of individual points from the cloud to defect areas with [email protected] of 87.5%. These indicators cannot be directly interpreted to assess the quality of the defect detection and classification system. Therefore, in the discussion paragraph, we describe the full-scale experiment we conducted to assess the quality of the system operation.

6. Discussion

During the study and assembly of the prototype, we identified limitations in the use of the proposed software and hardware platform for the implementation of the visual inspection of girth welds in steel pipelines. Among them are the limitations determined by the very procedure for checking the quality of the work performed, which consist of the range of diameters of the investigated pipes and the need for a uniform movement of the trolley with on-board equipment along the surface of the object under study at a low speed. The main disadvantage of the proposed approach is an incomplete list of the types of defects identified. However, this method of nondestructive testing does not involve the determination of such defects as internal porosity, internal slag inclusion, tungsten inclusion, or elongated indication and undercutting. Determining these types of defects certainly plays an important role in the safer operation of the pipeline line; however, it requires the implementation of expensive and resource-intensive methods; therefore, in the context of this work, we present our result in the field of a preliminary assessment of installation work performed directly in the construction process. The hardware of the proposed monitoring tool is characterized by the low cost of the individual components, which guarantees their easy replacement in case of failure, and low power consumption, which guarantees low installation dimensions with sufficient battery life. The software implements a system to determine and classify defects, as well as generates final reports on the inspection carried out with a three-dimensional reconstruction of the weld surface under study. It is based on a set of methods and approaches tested by other researchers, with the introduction of the authors’ corrections regarding the problem being solved and the area of research under consideration. The use of computer vision algorithms, machine learning models, and neural networks makes it possible to organize a procedure for the automated detection of defects in the test object with sufficient sensitivity and accuracy, which contributes to the application of this tool in practice. Based on the results of this study, 100 full-scale experiments were carried out to control the quality of welded joints with the involvement of a human expert from a certified laboratory of nondestructive testing methods. All experiments were carried out in clear sunny weather during the first half of the day at one site of installation working on laying a gas pipeline. As a result, the expert identified two serious violations of installation technology, which cast doubt on the possibility of further operation of the line with the assignment of the required category of the facility, and about seven minor defects that allow further use of the pipeline. The system identified approximately ten cases of violations in the production of pipeline welds, two of which were classified as potentially dangerous by the expert, and four that were classified as minor. The expert marked the remaining four detections as false positives; in addition, he noted three not considered by the system. Unfortunately, at the time of writing this article, there were no other objects for nondestructive testing of butt joints of steel pipelines in the work of LLC «Cyber Vision», and we were unable to expand the statistical sample of the actual system performance.

7. Conclusions

In, the proposed study several results were obtained. We assembled a working prototype of a hardware platform based on Raspberry Pi modules and expansion boards. Several scenarios for data preprocessing and training sets formation were proposed and successfully implemented, allowing extrapolating, and at some stages, the available X-ray images of welds to detect and classify defects based on extracted frames from a conventional video sequence of the visual nondestructive testing method. To implement the system for detecting and classifying defects, the application of the system of computer vision algorithms and the machine learning model was chosen, which provides the prediction fusion at the deliberative procedure level. The trained models received the following indicators: The YOLOv5 first case trained on weld contour images extracted from radiographic and conventional images of the test object achieved a [email protected] value of 86.9%. The second case YOLOv5, trained on radiographic inspection images with no markings on internal defects and retrained on conventional photographs of defective welds, received [email protected] 96.8%. Random forest, which determines the probability that a point belongs to a defective zone from the weld surface cloud points, reached [email protected] 87.5%. In addition, we conducted a full-scale experiment in the field, in which the proposed hardware and software platform showed its stability. We fully agree that our proposed approach is inferior to existing solutions based on radiographic and ultrasound examinations. However, the goal of our work was to create a convenient and fast method for a preliminary assessment of the quality of welding work “in the field.” In most cases, this will have a significant economic and temporary effect in the case of early detection of weld defects. In addition, we outlined a further trajectory of work in this direction. It is planned to consider the capabilities of Inceptionv4 in solving the identified problems, as well as to adjust the YOLOv5 structure by adding convolution from GoogLeNet/Inception to it in order to improve the accuracy and speed of defect detection. Additionally, among other things, the performance of the Raspberry Pi4 node in some tasks does not meet expectations, so we plan to select another hardware platform in accordance with our vision of an economical product.

Author Contributions

Conceptualization, A.V.C. and M.A.B.; data curation, S.A.P.; formal analysis, A.A.A. and V.S.I.; funding acquisition, M.A.B.; investigation, O.O.K., A.V.C. and A.A.A.; methodology, O.O.K. and A.V.C.; project administration, M.A.B.; resources, V.S.I. and S.A.P.; software, A.A.A., D.S.P., V.S.I. and S.A.P.; supervision, A.V.C.; validation, A.A.A. and D.S.P.; visualization, D.S.P.; writing—original draft, A.A.A. and D.S.P.; writing—review and editing, O.O.K. and A.V.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by State assignment no. 0852-2020-0019 to the Southern Federal University.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

Research was financially supported by the Ministry of Science and Higher Education of the Russian Federation (State assignment in the field of scientific activity, No. 0852-2020-0019).

Conflicts of Interest

The authors declare no conflict of interest. The founding sponsors had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

References

  1. Csontos, B.; Halász, L.; Heckl, I. Event-Driven Simulation Method for Fuel Transport in a Mesh-like Pipeline Network. Comput. Chem. Eng. 2022, 157, 107611. [Google Scholar] [CrossRef]
  2. Johnstone, R.W. Transportation Systems and Security Risks. In Protecting Transportation; Elsevier: Amsterdam, The Netherlands, 2015; pp. 73–106. ISBN 978-0-12-408101-7. [Google Scholar]
  3. Das, D.; Das, S.K.; Parhi, P.K.; Dan, A.K.; Mishra, S.; Misra, P.K. Green Strategies in Formulating, Stabilizing and Pipeline Transportation of Coal Water Slurry in the Framework of WATER-ENERGY NEXUS: A State of the Art Review. Energy Nexus 2021, 4, 100025. [Google Scholar] [CrossRef]
  4. Umar, H.A.; Abdul Khanan, M.F.; Ogbonnaya, C.; Shiru, M.S.; Ahmad, A.; Baba, A.I. Environmental and Socioeconomic Impacts of Pipeline Transport Interdiction in Niger Delta, Nigeria. Heliyon 2021, 7, e06999. [Google Scholar] [CrossRef] [PubMed]
  5. Murrill, B.J. Pipeline Transportation of Natural Gas and Crude Oil: Federal and State Regulatory Authority; Library of Congress, Congressional Research Service: Washington, DC, USA, 2016. [Google Scholar]
  6. Petro, P.P. Study of Plastic vs. Steel Pipe Performance. Gas Dig. 1975, 1. Available online: https://www.osti.gov/biblio/7219274 (accessed on 24 July 2022).
  7. Sotoodeh, K. Piping and Valve Corrosion Study. In A Practical Guide to Piping and Valves for the Oil and Gas Industry; Elsevier: Amsterdam, The Netherlands, 2021; pp. 585–627. ISBN 978-0-12-823796-0. [Google Scholar]
  8. Annila, L. Nondestructive Testing of Pipelines; Springer: Berlin/Heidelberg, Germany, 2018. [Google Scholar]
  9. Becht, C. Process Piping: The Complete Guide to ASME B31.3, 3rd ed.; ASME Press: New York, NY, USA, 2009; ISBN 978-0-7918-0286-1. [Google Scholar]
  10. Carvalho, A.A.; Rebello, J.M.A.; Souza, M.P.V.; Sagrilo, L.V.S.; Soares, S.D. Reliability of Nondestructive Test Techniques in the Inspection of Pipelines Used in the Oil Industry. Int. J. Press. Vessel. Pip. 2008, 85, 745–751. [Google Scholar] [CrossRef]
  11. Lumb, R.F. Nondestructive Testing of High-Pressure Gas Pipelines. Nondestruct. Test. 1969, 2, 259–268. [Google Scholar] [CrossRef]
  12. Zhang, B.; Liu, F.; Liu, C.; Li, J.; Zhang, B.; Zhou, Q.; Han, X.; Zhao, Y. An Ultrasonic Nondestructive Testing Method for the Measurement of Weld Width in Laser Welding of Stainless Steel. In Proceedings of the AIP Conference Proceedings, Wuhan, China, 7–9 April 2017; p. 040082. [Google Scholar]
  13. Ma, Q.; Tian, G.; Zeng, Y.; Li, R.; Song, H.; Wang, Z.; Gao, B.; Zeng, K. Pipeline In-Line Inspection Method, Instrumentation and Data Management. Sensors 2021, 21, 3862. [Google Scholar] [CrossRef]
  14. Vishal, V.; Ramya, R.; Vinay Srinivas, P.; Vimal Samsingh, R. A Review of Implementation of Artificial Intelligence Systems for Weld Defect Classification. Mater. Today Proc. 2019, 16, 579–583. [Google Scholar] [CrossRef]
  15. Fioravanti, C.C.B.; Centeno, T.M.; De Biase Da Silva Delgado, M.R. A Deep Artificial Immune System to Detect Weld Defects in DWDI Radiographic Images of Petroleum Pipes. IEEE Access 2019, 7, 180947–180964. [Google Scholar] [CrossRef]
  16. Wang, Y.; Sun, Y.; Lv, P.; Wang, H. Detection of Line Weld Defects Based on Multiple Thresholds and Support Vector Machine. NDT E Int. 2008, 41, 517–524. [Google Scholar] [CrossRef]
  17. Cassels, B. Weld Defect Detection Using Ultrasonic Phased Arrays; University of Central Lancashire: Preston, UK, 2018. [Google Scholar]
  18. Ajmi, C.; Zapata, J.; Elferchichi, S.; Zaafouri, A.; Laabidi, K. Deep Learning Technology for Weld Defects Classification Based on Transfer Learning and Activation Features. Adv. Mater. Sci. Eng. 2020, 2020, 1574350. [Google Scholar] [CrossRef]
  19. Yang, D.; Cui, Y.; Yu, Z.; Yuan, H. Deep Learning Based Steel Pipe Weld Defect Detection. Appl. Artif. Intell. 2021, 35, 1237–1249. [Google Scholar] [CrossRef]
  20. Sizyakin, R.; Voronin, V.V.; Gapon, N.; Zelensky, A.; Pižurica, A. Automatic Detection of Welding Defects Using the Convolutional Neural Network. In Proceedings of the Automated Visual Inspection and Machine Vision III, Munich, Germany, 27 June 2019; Beyerer, J., Puente León, F., Eds.; SPIE: Munich, Germany, 2019; p. 14. [Google Scholar]
  21. Cui, W.; Wang, K.; Zhang, Q.; Zhang, P. A Recognition Algorithm to Detect Pipe Weld Defects. Teh. Vjesn. 2017, 24, 1969–1975. [Google Scholar] [CrossRef]
  22. Oh, S.; Jung, M.; Lim, C.; Shin, S. Automatic Detection of Welding Defects Using Faster R-CNN. Appl. Sci. 2020, 10, 8629. [Google Scholar] [CrossRef]
  23. Kumar, R.; Somkuva, V.; Nitttr, B. A Review on Analysis, Monitoring and Detection of Weld Defect Products. IJERT 2015, V4, IJERTV4IS110556. [Google Scholar] [CrossRef]
  24. Chen, P.; Li, R.; Fu, K.; Zhao, X. Research and Method for In-Line Inspection Technology of Girth Weld in Long-Distance Oil and Gas Pipeline. J. Phys. Conf. Ser. 2021, 1986, 012052. [Google Scholar] [CrossRef]
  25. Gaol, A.L.; Sutanto, H.; Basuki, W.W. Analysis of Welding Disabilities on Carbon Steel Pipes with SMAW Reviewed from Radiography Test Results. Int. J. Appl. Eng. Res. 2019, 14, 7. [Google Scholar]
  26. Moghaddam, A.A.; Rangarajan, L. Classification of Welding Defects in Radiographic Images. Pattern Recognit. Image Anal. 2016, 26, 54–60. [Google Scholar] [CrossRef]
  27. Kumaresan, S.; Aultrin, K.S.J.; Kumar, S.S.; Anand, M.D. Transfer Learning with CNN for Classification of Weld Defect. IEEE Access 2021, 9, 95097–95108. [Google Scholar] [CrossRef]
  28. Chady, T.; Sikora, R.; Misztal, L.; Grochowalska, B.; Grzywacz, B.; Szydłowski, M.; Waszczuk, P.; Szwagiel, M. The Application of Rough Sets Theory to Design of Weld Defect Classifiers. J. Nondestruct. Eval. 2017, 36, 40. [Google Scholar] [CrossRef]
  29. Chang, J.; Kang, M.; Park, D. Low-Power On-Chip Implementation of Enhanced SVM Algorithm for Sensors Fusion-Based Activity Classification in Lightweighted Edge Devices. Electronics 2022, 11, 139. [Google Scholar] [CrossRef]
  30. Tarek, H.; Aly, H.; Eisa, S.; Abul-Soud, M. Optimized Deep Learning Algorithms for Tomato Leaf Disease Detection with Hardware Deployment. Electronics 2022, 11, 140. [Google Scholar] [CrossRef]
  31. Meivel, S.; Indira Devi, K.; Muthamil Selvam, T.; Uma Maheswari, S. Real Time Analysis of Unmask Face Detection in Human Skin Using Tensor Flow Package and IoT Algorithm. Mater. Today Proc. 2021, S2214785320405826. [Google Scholar] [CrossRef]
  32. Abed, A.M.; Gitaffa, S.A.; Issa, A.H. Robust Geophone String Sensors Fault Detection and Isolation Using Pattern Recognition Techniques Based on Raspberry Pi4. Mater. Today Proc. 2021, S2214785321032806. [Google Scholar] [CrossRef]
  33. Asanza, V.; Avilés-Mendoza, K.; Trivino-Gonzalez, H.; Rosales-Uribe, F.; Torres-Brunes, J.; Loayza, F.R.; Peláez, E.; Cajo, R.; Tinoco-Egas, R. SSVEP-EEG Signal Classification Based on Emotiv EPOC BCI and Raspberry Pi. IFAC-Pap. 2021, 54, 388–393. [Google Scholar] [CrossRef]
  34. Gaggion, N.; Ariel, F.; Daric, V.; Lambert, É.; Legendre, S.; Roulé, T.; Camoirano, A.; Milone, D.H.; Crespi, M.; Blein, T.; et al. ChronoRoot: High-Throughput Phenotyping by Deep Segmentation Networks Reveals Novel Temporal Parameters of Plant Root System Architecture; Plant Biology: Portland, OR, USA, 2020. [Google Scholar]
  35. Rouhi, A.H.; Thom, J.A. Encoder Settings Impact on Intra-Prediction-Based Descriptors for Video Retrieval. J. Vis. Commun. Image Represent. 2018, 50, 263–269. [Google Scholar] [CrossRef]
  36. Appiah, O.; Asante, M.; Hayfron-Acquah, J.B. Improved Approximated Median Filter Algorithm for Real-Time Computer Vision Applications. J. King Saud Univ. Comput. Inf. Sci. 2020, 34, 782–792. [Google Scholar] [CrossRef]
  37. Mery, D.; Riffo, V.; Zscherpel, U.; Mondragón, G.; Lillo, I.; Zuccar, I.; Lobel, H.; Carrasco, M. GDXray: The database of X-ray images for nondestructive testing. J. Nondestruct. Eval. 2015, 4, 1–12. [Google Scholar] [CrossRef]
  38. Mohamed, E.; Shaker, A.; El-Sallab, A.; Hadhoud, M. INSTA-YOLO: Real-Time In-stance Segmentation. arXiv 2021, arXiv:2102.06777. [Google Scholar]
  39. Liao, L.; Tang, S.; Liao, J.; Li, X.; Wang, W.; Li, Y.; Guo, R. A Supervoxel-Based Random Forest Method for Robust and Effective Airborne LiDAR Point Cloud Classification. Remote Sens. 2022, 14, 1516. [Google Scholar] [CrossRef]
  40. Xue, D.; Cheng, Y.; Shi, X.; Fei, Y.; Wen, P. An Improved Random Forest Model Applied to Point Cloud Classification. IOP Conf. Ser. Mater. Sci. Eng. 2020, 768, 072037. [Google Scholar] [CrossRef]
  41. Zeybek, M. Classification of UAV Point Clouds by Random Forest Machine Learning Algorithm. Turk. J. Eng. 2021, 5, 48–57. [Google Scholar] [CrossRef]
  42. Geiger, A.; Roser, M.; Urtasun, R. Efficient Large-Scale Stereo Matching. In Computer Vision—ACCV 2010; Kimmel, R., Klette, R., Sugimoto, A., Eds.; Lecture Notes in Computer Science; Springer: Berlin/Heidelberg, Germany, 2011; Volume 6492, pp. 25–38. ISBN 978-3-642-19314-9. [Google Scholar]
  43. Kaczmarek, A.L. Stereo Vision with Equal Baseline Multiple Camera Set (EBMCS) for Obtaining Depth Maps of Plants. Comput. Electron. Agric. 2017, 135, 23–37. [Google Scholar] [CrossRef]
  44. Chmelar, P.; Beran, L.; Rejfek, L. The Depth Map Construction from a 3D Point Cloud. MATEC Web Conf. 2016, 75, 03005. [Google Scholar] [CrossRef]
  45. Whebell, R.M.; Moroney, T.J.; Turner, I.W.; Pethiyagoda, R.; McCue, S.W. Implicit Reconstructions of Thin Leaf Surfaces from Large, Noisy Point Clouds. Appl. Math. Model. 2021, 98, 416–434. [Google Scholar] [CrossRef]
  46. Wu, H.; Xu, Z.; Liu, C.; Akbar, A.; Yue, H.; Zeng, D.; Yang, H. LV-GCNN: A Lossless Voxelization Integrated Graph Convolutional Neural Network for Surface Reconstruction from Point Clouds. Int. J. Appl. Earth Obs. Geoinf. 2021, 103, 102504. [Google Scholar] [CrossRef]
Sensors 22 06201 g001 550
Figure 1. Connection diagram for Raspberry Pi4 modules and expansion boards.
Figure 1. Connection diagram for Raspberry Pi4 modules and expansion boards.
Sensors 22 06201 g001
Sensors 22 06201 g002 550
Figure 2. General network architecture of used YOLOv5.
Figure 2. General network architecture of used YOLOv5.
Sensors 22 06201 g002
Sensors 22 06201 g003 550
Figure 3. General view of the process of restoring the depth of an object based on a binocular approach.
Figure 3. General view of the process of restoring the depth of an object based on a binocular approach.
Sensors 22 06201 g003
Sensors 22 06201 g004 550
Figure 4. Results of preprocessing images of welded joints radiographic inspection.
Figure 4. Results of preprocessing images of welded joints radiographic inspection.
Sensors 22 06201 g004
Sensors 22 06201 g005 550
Figure 5. Samples of marked welds in steel pipelines images.
Figure 5. Samples of marked welds in steel pipelines images.
Sensors 22 06201 g005
Sensors 22 06201 g006 550
Figure 6. Stages of preliminary processing of image data.
Figure 6. Stages of preliminary processing of image data.
Sensors 22 06201 g006
Sensors 22 06201 g007 550
Figure 7. Examples of preprocessed and labeled image data.
Figure 7. Examples of preprocessed and labeled image data.
Sensors 22 06201 g007
Sensors 22 06201 g008 550
Figure 8. First case model training and validation metrics for a set of counter images.
Figure 8. First case model training and validation metrics for a set of counter images.
Sensors 22 06201 g008
Sensors 22 06201 g009 550
Figure 9. Mutual dependencies of the model metric indicators used (a) precision–confidence, (b) precision–recall, (c) recall–confidence, and (d) F1–confidence.
Figure 9. Mutual dependencies of the model metric indicators used (a) precision–confidence, (b) precision–recall, (c) recall–confidence, and (d) F1–confidence.
Sensors 22 06201 g009
Sensors 22 06201 g010 550
Figure 10. Second case model training and validation metrics for a set of counter images. (a) Correlogram of ROI labels; (b) Distribution density of image labeling parameters; (c) Confusion matrix; (d) Standard set YOLOv5 training and validation metrics.
Figure 10. Second case model training and validation metrics for a set of counter images. (a) Correlogram of ROI labels; (b) Distribution density of image labeling parameters; (c) Confusion matrix; (d) Standard set YOLOv5 training and validation metrics.
Sensors 22 06201 g010
Sensors 22 06201 g011 550
Figure 11. Mutual dependencies of the model metric indicators used in the second case: (a) precision–recall, (b) recall–confidence, (c) precision–confidence, and (d) F1–confidence.
Figure 11. Mutual dependencies of the model metric indicators used in the second case: (a) precision–recall, (b) recall–confidence, (c) precision–confidence, and (d) F1–confidence.
Sensors 22 06201 g011
Sensors 22 06201 g012 550
Figure 12. (a) Average ROC (b) ROC area to multilabel classification, (c) PRA map showing dependencies average precision, recall, and F1-score metrics. (d) Confusion matrix.
Figure 12. (a) Average ROC (b) ROC area to multilabel classification, (c) PRA map showing dependencies average precision, recall, and F1-score metrics. (d) Confusion matrix.
Sensors 22 06201 g012
Table
Table 1. Data structure of the point clouds of the weld surface of the steel pipeline.
Table 1. Data structure of the point clouds of the weld surface of the steel pipeline.
xyznxnynzRedGreenBlueAlpha
0−0.276752.15163.56214.5206034.282893.835602047103255
1−0.277502.15573.56644.7493003.107603.18440363596255
2−0.282502.16023.57013.1592002.989904.152304551110255
3−0.289002.16973.57094.0629004.117802.99978234999255
4−0.289472.16973.57184.1312004.567194.368101937106255
Table
Table 2. Values of YOLOv5 first case hyperparameters during training.
Table 2. Values of YOLOv5 first case hyperparameters during training.
ParameterMeaning
Number of epochs50
Batch size64
Learning rate0.01
Momentum0.937
Weight decay0.005
Image cachingYes
Table
Table 3. Values of YOLOv5 s case hyperparameters during training.
Table 3. Values of YOLOv5 s case hyperparameters during training.
ParameterMeaning
Number of epochs25
Batch size64
Learning rate0.01
Momentum0.937
Weight decay0.005
Image cachingYes
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Kartashov, O.O.; Chernov, A.V.; Alexandrov, A.A.; Polyanichenko, D.S.; Ierusalimov, V.S.; Petrov, S.A.; Butakova, M.A. Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection. Sensors 2022, 22, 6201. https://doi.org/10.3390/s22166201

AMA Style

Kartashov OO, Chernov AV, Alexandrov AA, Polyanichenko DS, Ierusalimov VS, Petrov SA, Butakova MA. Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection. Sensors. 2022; 22(16):6201. https://doi.org/10.3390/s22166201

Chicago/Turabian Style

Kartashov, Oleg O., Andrey V. Chernov, Alexander A. Alexandrov, Dmitry S. Polyanichenko, Vladislav S. Ierusalimov, Semyon A. Petrov, and Maria A. Butakova. 2022. "Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection" Sensors 22, no. 16: 6201. https://doi.org/10.3390/s22166201

APA Style

Kartashov, O. O., Chernov, A. V., Alexandrov, A. A., Polyanichenko, D. S., Ierusalimov, V. S., Petrov, S. A., & Butakova, M. A. (2022). Machine Learning and 3D Reconstruction of Materials Surface for Nondestructive Inspection. Sensors, 22(16), 6201. https://doi.org/10.3390/s22166201

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop