JP7422023B2 - X-ray image processing device and X-ray image processing method - Google Patents

Description

The present invention relates to an X-ray image processing device and an X-ray image processing method.

X-ray baggage inspection devices are used for baggage inspection at airports, large-scale event venues, and the like. Generally, an X-ray baggage inspection device generates a grayscale image showing the amount of X-ray transmission, and a color image in which the material is determined and colored for each material. The general operation is for an inspector to visually check the images to confirm the presence of dangerous objects, and if the inspector finds a dangerous object, the bag is opened and inspected.

Highly trained inspectors are required to confirm whether X-ray images contain hazardous materials. Therefore, it is difficult to temporarily secure a large number of inspectors for a large-scale event, for example, in terms of prior training and cost. Therefore, attempts are being made to automate the detection of hazardous materials in order to reduce the burden on inspectors as much as possible.

One way to automate image recognition is to use image recognition technology that utilizes deep learning using AI (artificial intelligence). Deep learning is widely used in applications such as video analysis, and is becoming more popular because it provides high recognition accuracy. For example, Patent Document 1 discloses a method of obtaining high recognition accuracy by combining material density and shape recognition results.

JP 2018-4363 Publication

The technique described in Patent Document 1 targets images taken from one direction with an X-ray inspection device. That is, since one recognition result is obtained for one photographing direction, if there are a plurality of photographing directions, the same number of recognition results as there are photographing directions can be obtained. Therefore, the inspector has to confirm multiple recognition results, which poses a problem in that the inspector's confirmation time increases. Furthermore, images taken from one direction have a small amount of information, and there is a problem in that the recognition accuracy is lower than that of a CT method that can photograph an article in 3D (three dimensions).

On the other hand, X-ray inspection devices that can take images from two directions are becoming more popular because they are cheaper than CT systems. An X-ray inspection device that can take images from two directions can detect the knife by taking images of the object from two directions, even if the object is thin, such as a knife. It becomes possible to clearly distinguish the shape. Therefore, by checking the article from two directions, it is possible to prevent items from being overlooked more than when checking the article from one direction. Even in the recognition of prohibited items using AI, it is considered a good idea to integrate the recognition results from two directions and then present the final recognition result.

An object of the present invention is to provide an X-ray image processing device and an X-ray image processing method that can improve recognition accuracy of prohibited items in an X-ray inspection device that can take images from a plurality of directions.

In order to solve the above problems, one of the representative X-ray image processing apparatuses of the present invention includes an X-ray image acquisition section that acquires a plurality of X-ray images taken of an article from a plurality of directions; An article recognition section that recognizes an article using a learning model that has learned the first side of the article for a line image, and a recognition result integration section that integrates recognition results for multiple X-ray images. It has a screen generation unit that generates screen information based on the recognition result.

According to the present invention, it is possible to provide an X-ray image processing device and an X-ray image processing method that can improve recognition accuracy of prohibited items in an X-ray inspection device that can take images from a plurality of directions.
Problems, configurations, and effects other than those described above will be made clear by the following description of the embodiments.

FIG. 1 is a configuration diagram of an X-ray inspection device. 1 is a functional configuration diagram of an X-ray image processing device 102. FIG. 3 is a flowchart showing a processing operation for recognizing and displaying an article. FIG. 3 is a diagram illustrating an example of dividing image data into luggage units. It is a figure which shows the example of the image of the front and the side of an article. FIG. 3 is a diagram showing an example of screen generation. FIG. 3 is a diagram showing an example of screen generation. FIG. 3 is a diagram showing an example of screen generation.

Preferred embodiments of the X-ray image processing apparatus and X-ray image processing method of the present invention will be described below.
In this disclosure, the "front" of an article refers to the side that is visible from the direction that makes it easy for a general inspector to identify the article, and the "side" of the article refers to the direction in which the front is visible. It means a surface that can be seen from the vertical direction , and the "reliability" of a recognition result means the likelihood of the recognition result .

FIG. 1 is a configuration diagram of an X-ray inspection apparatus having an X-ray image processing device.
The X-ray inspection device 100 is a device widely used as a baggage inspection device in airport security inspections, for example, and includes an X-ray device main body (hereinafter referred to as the device main body) 101, an X-ray image processing device 102, and a display section. 103 and an input section 104. The X-ray image processing device 102 is, for example, a personal computer (PC). The display unit 103 may be configured with two or more displays, such as when displaying images taken from two directions, or when displaying a color image and a grayscale image.

The device main body 101 has an irradiation mechanism that irradiates X-rays and an X-ray imaging mechanism that measures the amount of X-rays transmitted by photographing objects such as baggage. (sometimes referred to as "transmission amount data" or "transmission amount"). The X-ray image processing device 102 determines whether the baggage is safe based on the article recognition result of the X-ray image taken by the device main body 101. The X-ray image processing apparatus of this embodiment has a learning function and can learn a model for image recognition using accumulated article information. The display unit 103 is a display terminal that displays an X-ray image, and the inspector can visually check the X-ray image displayed on the screen.

The apparatus main body 101 has a conveyance mechanism including a belt conveyor for conveying cargo, and the conveyance mechanism is controlled by a control unit to drive and stop the belt conveyor. If the X-ray image processing device 102 determines that the item is dangerous baggage (alert object) as a result of object recognition, it lights up the device main body 101 or an indicator lamp installed near the device main body 101, indicating that the object is an alert object. Notify the inspector that this is the case. Note that the display unit 103 or the audio output unit of the X-ray image processing device 102 may be used to notify that the object is an alert target.

Two types of X-ray sensors (hereinafter simply referred to as "sensors") that measure the amount of transmitted X-rays are arranged in the transport mechanism of this embodiment, and two types of data are acquired. That is, one sensor acquires low energy data and another sensor acquires high energy data. The X-ray image processing device 102 determines the material of the object based on the difference between high-energy data and low-energy data acquired by the two sensors. The X-ray sensor only needs to be able to acquire X-ray data from which the material can be determined, and the detection method of the X-ray sensor does not matter. For example, a backscatter type material determination sensor or the like may be used, or other sensors may be used. Furthermore, in the X-ray inspection device of this embodiment, sets of the above-mentioned sensors (two types of X-ray sensors) are installed at multiple locations, such as the side (vertical surface) and ceiling surface (horizontal surface) of the area where the baggage passes. It is being For example, an X-ray inspection device that can take images from two directions can simultaneously take images of luggage from two directions: top and side. Usually, a line sensor is used as the sensor. In such a case, in order to avoid interference between the plurality of X-ray light sources, the first set of sensors and the second set of sensors may be installed with their positions slightly shifted from each other in the cargo traveling direction.

FIG. 2 is a functional configuration diagram of the X-ray image processing apparatus 102.
The X-ray image processing device 102 includes a processing unit (CPU: Central Processing Unit) 201, a main memory 202, a display interface (I/F) unit 203 that connects the display unit 103, and an input interface (I/F) that connects the input unit 104. F) It is an information processing device having a section 204, a communication section 205, and a storage section 210.

The processing unit 201 executes a program to realize predetermined functions and operations. The storage unit 210 stores an X-ray image acquisition program 212, an article recognition program 213, a recognition result integration program 214, a screen generation program 215, a calibration program 216, an article learning program 217, X-ray image data 218, and a learning model 219. .

Here, the X-ray image acquisition program 212 acquires transmission amount data of X-rays photographed by the apparatus main body 101 from a dedicated interface or a general-purpose screen output terminal such as a VGA terminal. Note that in the case of an X-ray inspection device used at an airport security checkpoint, images generally taken from two directions are often displayed on two independent screens. In that case, the X-ray transmission amount data may be obtained from a plurality of input terminals. Details of the operation by the X-ray image acquisition program 212 will be described later.

The article recognition program 213 generates a color image that visualizes the material information of the baggage and the density information of the article generated from the transmission amount data of the two types of X-ray sensors photographed by the device main body 101, or a color image that visualizes the material information of the baggage and the density information of the article. Using deep learning segmentation technology, objects contained in the image are recognized pixel by pixel using grayscale images obtained by converting sensor transmission amount data. Details of the operation by the article recognition program 213 will be described later.

The recognition result integration program 214 integrates the recognition results of two images taken from two directions and recognizes the article with high precision. Furthermore, when presenting one screen to the inspector, the image to be presented to the inspector is selected from two images taken from two directions. Details of the operation of the recognition result integration program 214 will be described later.

The screen generation program 215 generates screen information to be displayed on the monitor of the inspector terminal based on the recognition results integrated by the recognition result integration program 214, and displays the generated screen information on the display unit 103. Details of the operation of the screen generation program 215 will be described later.

The calibration program 216 adjusts the positions of images so that the positions of a plurality of images taken from a plurality of directions can be associated with each other. Details of the operation of the calibration program 216 will be described later.

The article learning program 217 performs learning using deep learning segmentation processing or the like on the input color image or grayscale image, and generates a learning model 219 .

As the X-ray image data 218, a color image or a grayscale image indicating material information and density information is registered from X-ray transmission amount data measured by two types of sensors. For color images, the material of the article is determined from the difference between the X-ray transmission amount data of two types of sensors. A color image that is (light) is registered. Note that if the sensor is installed for imaging in two directions, two images from different directions are taken in one imaging and are registered in the X-ray image data 218.

Note that the above-mentioned X-ray image acquisition program 212 , article recognition program 213 , recognition result integration program 214 , screen generation program 215 , calibration program 216 , and article learning program 217 stored in the storage unit 210 are stored in the processing unit 201 . When executed, the functions of the X-ray image acquisition section 212', article recognition section 213', recognition result integration section 214', screen generation section 215', calibration section 216', and article learning section 217' are performed respectively. Realize.

The X-ray image data 218 is a color image or a grayscale image of the baggage taken by the X-ray inspection apparatus 100. For example, if the X-ray inspection device 100 is a device that can take images from two directions, vertical and horizontal, two images, one in the vertical direction and the other in the horizontal direction, are stored as X-ray image data 218 for one baggage along with the baggage ID. recorded in the area.

In the learning model 219, parameter information of the model learned by the article learning program 217 is recorded. Note that, as the learning model, learning models that are trained on one article in various directions such as the front and the side may be used. However, in this example, it is possible to determine not only what the item is, but also whether it is the front or side view of the item based on the image of the item to which the learning model is applied. As a learning model used for this, for example, a learning model may be used in which a front image and a side image of a smartphone are learned as images of different items.
By using this kind of learning model, in cases where the front and side surfaces look very different, such as smartphones, the front and side surfaces are learned as different objects, which allows for more accurate identification of the front and side surfaces. A learning model can be constructed based on the

Next, processing operations for recognizing and displaying articles will be described with reference to FIG. This processing operation is a processing operation during inspection work using the X-ray inspection apparatus 100.

(S301)
First, the X-ray image acquisition program 212 acquires RAW data, which is transmission amount data acquired by an X-ray sensor, or an X-ray image (screen signal displayed on a display) in which the transmission amount data is visualized. Here, if the input is RAW data of two types of energy, high and low, the material of each pixel of the image is determined from the difference information of the transmission amount data of two types of energy, high and low. The material can be determined by, for example, a method widely known in this field that classifies the material into four types: metal, inorganic material, organic material, and others, based on differential information of two types of energy transmission, high and low. Then, using the material information and the amount of X-ray transmission of either high or low energy, the material information is the color, and the amount of transmission is the darkness (areas with high density are dark, areas with low density are light). Generate an image.

Here, if the image data acquired from the device main body 101 is not divided for each package to be photographed, that is, if multiple packages are reflected in one image data, the first division shown below is performed. The image data is divided into parcel units using the third dividing method.

The first division method is a method in which a screen signal to be displayed on a display is input from a VGA output terminal of the device main body 101 or the like. First, a screen signal is acquired, and the degree of change in the frame image is used to determine whether the displayed content of the package has changed. When the photograph of the baggage is finished, the baggage is displayed stationary on the screen. If the dark part where the amount of transmission is below the threshold value is defined as the baggage area, it is possible to specify the end of photographing the baggage and the baggage area at the end of the photographing based on whether or not the baggage area changes.

An example of the second division method is shown in FIG. A screen signal or RAW data is acquired, and an integral value of the amount of transmission is determined for every fixed number of lines or every fixed period of time. It is determined that there is baggage in areas where the amount of transmission is small (=dark), and it is determined that there is no baggage in areas where the amount of transmission is large (=thin), and the image is divided into baggage units. If a single line is used to judge, there is a possibility of erroneous judgment due to noise, etc. Therefore, if the integral value of the amount of transmission obtained by taking the moving average is less than the threshold value, there is baggage, and if it is above the threshold value, there is no baggage. It is also possible to judge. Further, since a thin part may be included in the middle of the baggage, it may be determined that the thin part is a space between two pieces of baggage if the thin part continues for a certain amount. In the graph of FIG. 4, the smaller the amount of transmission (=darker), the higher the plot, and the larger the amount of transmission (=lighter), the lower the plot, and the numbers indicate what line the line is.

The third dividing method is a method using a baggage detection sensor installed at the baggage input port of the X-ray inspection device. In a typical X-ray inspection device, a sensor capable of detecting the shielding is installed at a position where the light beam is shielded by the baggage, for example. By adding the time from when the sensor detects the baggage until the leading edge of the baggage arrives at the line sensor to the time when the sensor detects the baggage, the time at the baggage start point (shooting start time) can be obtained. Also, by adding the time from the time when the sensor stopped detecting the baggage until the end of the baggage arrives at the line sensor to the time when the sensor stopped detecting the baggage, the time of the baggage end point (shooting end time) can be obtained. By acquiring the time of the baggage start point and the time of the baggage end point, if the input is a screen signal, the baggage area can be specified from the screen of the time of the baggage end point and the information of the photographing time. If the input is RAW data, an image of the luggage area can be generated by using the RAW data from the start time to the end time as the RAW data of one luggage.

In an X-ray inspection apparatus that can take images from two directions, the above dividing method is applied to, for example, an image taken from the vertical direction and an image taken from the horizontal direction.

(S302)
Next, the calibration program 216 associates the photographed images in the two directions acquired by the X-ray image acquisition program 212 with the position in the traveling direction of the luggage. For example, when the X-ray image acquisition program 212 acquires two X-ray images in two directions, the appearance of the article differs depending on the photographing direction, and the position of the line sensor differs. In many cases, the correspondence between pixel-by-pixel positional relationships is not clear. In general, when inspectors visually check images from inspection equipment, or when applying AI-based product recognition to each image, it is not necessary to correlate the two images pixel by pixel, so the inspector's terminal The images displayed on two screens in two directions may be displayed shifted by the distance between the line sensors. Therefore, using the calibration program 216, correspondence between the pixels of the two images is performed using the first to third calibration methods described below.

The first calibration method uses time intervals between line sensors. For example, in the case of a method using RAW data, there is a certain time lag between data input from two line sensors. Therefore, the data input time is adjusted by the time interval between line sensors so that the times are synchronized on a line-by-line basis, thereby establishing correspondence between pixels.

The second calibration method is a calibration method in which a screen signal is input and the first image division method or the second image division method is used in the step (S301) of acquiring an X-ray image. When using an image division method based on the amount of transmission (density), images taken from two directions have different distributions of transmission, so the same position may not necessarily be determined as the starting point and ending point. is not limited. In addition, in the case of an image division method that cuts out only the baggage area, it is assumed that the margins, that is, the parts with a large amount of transparency, will be deleted not only in the direction of travel of the baggage, but also in the direction of travel and perpendicularly, so images can be taken from two directions. The images cut out into individual parcels have different resolutions and aspect ratios.

Therefore, the traveling direction of the luggage is correlated using the time interval between the two line sensors and the photographing start time and photographing end time of the two images. A margin image is added to the image with a later shooting start time to match the image with an earlier shooting start time, and a margin image is added to the image with an earlier shooting end time to match the image with a later shooting end time. do. Note that in the direction perpendicular to the traveling direction of the baggage, the size of the baggage is different in the vertical direction and the horizontal direction, and it is difficult to make a pixel-by-pixel correspondence, so the sizes of the images may be different. That is, it is guaranteed that the image adjusted by the calibration always captures the same article at the same position with respect to the traveling direction of the baggage.

The third calibration method measures the time interval between the line sensors used in the first calibration method or the second calibration method by photographing a sample baggage with a specific pattern at the beginning of using the device. It's a method. This sample baggage contains patterns made with lead ink that can be seen in X-ray images, and whose location can be determined when photographed. The time interval between line sensors can be measured based on the difference between the photographing time of the image pattern photographed first and the photographing time of the image pattern photographed later. Even if the distance between the line sensors is fixed, if an error occurs in the speed at which the belt conveyor moves, accurate positional correspondence may not be possible at the time interval set at the time of initial installation of the device. Therefore, by periodically correcting errors using the third calibration method, accurate correspondence can be achieved.

By performing calibration, two images taken from two directions can be displayed with their positions aligned in the traveling direction of the luggage. Additionally, two images taken from two directions can be comprehensively recognized using AI.

Note that the acquired X-ray image data is registered in the X-ray image data 218 because it is used in step S303 for the next article recognition or when the inspector refers to past image data.

(S303)
Next, the article recognition program 213 acquires the two images in the two directions acquired in step S302 from the X-ray image data 218, and recognizes the article pixel by pixel using the learning model 219 and deep learning segmentation processing. . Here, a library such as "Fully Convolutional Instance-aware Semantic Segmentation" or "Mask R-CNN" which is widely known as an OSS (open source software) library can be used to recognize the article. Note that the recognition method does not matter as long as the article can be identified in pixel units.

Here, since there are two images taken in two directions, recognition of the article is performed on two images, an image taken from the first direction and an image taken from the second direction. For example, if the front of the smartphone is photographed in the first direction, the side of the smartphone will be photographed in the second direction. In the case of a smartphone, it is easy to distinguish that it is a smartphone from an image taken from the front, but it is difficult to distinguish from an image taken from the side. Furthermore, it is also difficult to distinguish between articles when there are overlapping articles in the photographed direction. In this way, since a recognition result may be obtained for only one image, the recognition results for the two captured images are often different.

Furthermore, when recognizing an object, by using a model that has been trained as separate models for the front and side, it is possible to simultaneously determine whether the object is close to the front or the side. Can be done. Here, if the photograph is taken from an angle between the front and the side, it is better to choose the one that is closer between the front and the side of the article. In addition, based on the recognition results using a learning model that combines the front and side views, other methods are used, such as the area of the item and the degree of similarity by template matching with sample images of the front and side views prepared in advance. It does not matter whether it is the front or the side. In other words, it does not matter whether it is determined as smartphone (front) or smartphone (side) at the time of recognition, or it may be determined whether it is front or side based on the recognition result using a learning model that combines front and side. do not have.

Note that in the description of this embodiment, the description is given in two directions to make the explanation easy to understand, but some articles may appear differently when photographed from three or more directions. If the image looks different in three or more directions, directions other than two may be added.

In addition, a learning model trained using a color image and a learning model trained using a grayscale image are prepared in the learning model 219, and when there is a large overlap between articles, that is, when the area of a dark part of the image is larger than a threshold value, etc. It is also possible to perform recognition using a learning model trained on grayscale images. This makes it possible to avoid the influence of changes in color indicating the type of material due to overlapping articles.

(S304)
Next, the recognition result integration program 214 performs an integration process of merging or selecting recognition results for the two-way recognition results obtained in step S303 of recognizing the article, using the first integration method to third integration method described below. This will be done using an integrated method. In the step of recognizing the article, the reliability of the recognition result is obtained together with the recognition result of the article. That is, when recognition is attempted using a learning model for the front of the article and a learning model for the side of the article for each of images in two directions, the respective recognition results and the reliability of the recognition results are obtained. Integration processing is performed using the obtained recognition results and the reliability of the recognition results.

The first integration method is an integration method that identifies an article when the front of the article and the side of the article can be recognized at the same position with respect to the traveling direction of images in two directions. If an object is recognized only when the front side of the object can be recognized in one direction and the side surface of the object can be recognized in the other direction, excessive detection (determining something that is not a prohibited object as a prohibited object) can be avoided. It can be suppressed. Furthermore, in order to prevent detection failures, the reliability threshold for employing the recognition results in each direction may be lowered. Even in this case, since judgment is made based on recognition in two directions, excessive detection can be suppressed. Therefore, there is an effect that it is possible to suppress over-detection while suppressing detection omissions. That is, in an X-ray inspection apparatus that takes images in two directions, object recognition is generally performed in each direction, but by using the present integration method, recognition can be performed in consideration of the 3D shape of the object.

Note that if the proportion of a portion with a small amount of transmission, such as an X-ray shielding object such as lead, in an image in any direction is equal to or greater than a threshold value, it is determined that the image does not include an article. Then, the reliability threshold for the other direction is increased, and the recognition result from the other direction is adopted. Therefore, if an object is visible in both directions, the recognition results from the two directions are integrated, and if the object is not visible in either direction due to an object blocking X-rays, The recognition result of one direction is adopted.

The second integration method is an integration method that identifies an article when either the front of the article or the side of the article can be recognized in an image in either direction. It is better than the first integration method because it does not recognize the article only when the front of the article can be recognized in one of the two directions and the side of the article can be recognized in the other. Although this increases the number of excessive detections, it is possible to prevent failure to detect prohibited items. For example, as shown in FIG. 5, even if a smartphone can be identified in an image viewed from the front, but cannot be identified as a smartphone in an image viewed from the side, it is still recognized as a smartphone. Then, the recognition result that the smartphone is in the same position in the direction of travel as the front-view image is added to the side-view image. Since it is difficult to accurately correlate the direction perpendicular to the direction of travel of the baggage, it indicates the range in which the goods may be present. This has the effect of preventing the inspector from overlooking the side image when looking at it.

The third integration method is a method of selecting the image in the direction in which the prohibited article is shown close to the front from among the images in the two directions. That is, of the images in the two directions, the image with the higher reliability of the recognition result for the front of the article is selected. If the inspector is checking only one monitor, he or she must select an image to display. In that case, it is preferable to present an image that more clearly shows the prohibited object. Therefore, an image that shows more prohibited items or an image that shows the front side of the prohibited item is selected. Which of the number of prohibited items and the front should be prioritized depends on the operational conditions, so it can be set in advance. This has the effect that even if the inspector only checks one monitor, he or she can check as many prohibited items as possible. In addition, in combination with the second integration method, if there is a prohibited item recognized only in the unselected image, the area where the prohibited item may be included in the selected image is indicated by the broken line in Figure 5. Feel free to add.

Note that the first to third integration methods may be used in combination with multiple integration methods or used alone, or in combination with other integration methods than the first to third integration methods. I don't mind if you do.

(S305)
Next, if the result of the recognition result integration in step S304 includes a prohibited article, the screen generation program 215 displays an image with the relevant portion highlighted on the display unit 103. Note that if the prohibited item is not included, the captured image may be displayed as is.

In step S305, using the results of the recognition result integration in step S304, screen display data is generated and displayed on the screen using the first to third screen generation methods described below.

The first screen generation method is a screen generation method that selects and displays an image in the direction in which the front of the prohibited article is shown, as shown in an example in FIG. If a plurality of items are included, the image in the direction to be prioritized is selected and displayed based on predetermined conditions such as the priority as a prohibited item and the number of recognized prohibited items. Generally, when checking AI recognition results on one screen, displaying one image rather than displaying multiple images can reduce the visual burden on the inspector. Therefore, when an image to be confirmed from an inspection perspective is displayed, and for articles that are not recognized in that image, the second integration method of recognition result integration in step S304 is used to display prohibited articles in the image to be confirmed. It is only necessary to ensure that areas that may be included are displayed.

As shown in FIG. 7, the second screen generation method is a screen generation method in which images taken in two directions are displayed on two screens so that one screen corresponds to one direction. It is desirable to use the second integration method of integrating the recognition results in step S304 for the two images on the two screens to be displayed, so that there will be no products for which an alert is displayed only on one screen. Thereby, it is possible to prevent omissions in inspection due to the inspector checking only one screen and forgetting to check the other screen.

As shown in FIG. 8, the third screen generation method is a method of displaying two images, which have been aligned through the calibration in step S302, along with the recognition results on the top and bottom or left and right sides of one screen. Since the inspector can confirm the presence or absence of prohibited items using images that are aligned in the traveling direction of the baggage, the inspector can judge or specify the prohibited item by combining the recognition results in the two directions.

Note that the screens according to the first screen display method to the third screen generation method may be provided separately from the standard screen output by the device main body 101, or may be provided in place of the standard screen. Furthermore, the method of displaying the X-ray image may be a method in which the screen scrolls in synchronization with the photographing of the baggage, which is used on a standard screen, or a method in which the image is switched for each baggage. Also, since both the front and side of the item can be recognized, it is determined that it is highly likely to be a prohibited item, or because only one of the front and side of the item can be recognized, it may not be a prohibited item. However, depending on the recognition result, such as deciding that it is better to check just in case, you can change the way the alert is sent by color, sound, etc.

Another display method is to not show the direction in which the prohibited item is clearly shown, which increases the nervousness of the inspector, and if the inspector mistakenly judges that the image is okay, an alert will be displayed and the prohibited item will be displayed. It can also be used for training inspectors by presenting images that clearly show the image.

Furthermore, if the item is determined to be a prohibited item in only one direction, information can be outputted on the screen or in audio to prompt the inspector to change the direction and take the image again. Then, the two-direction images taken again are determined using the four-direction images added to the first two-direction images, and if different sides of the item are recognized in at least two images, the item is classified as a prohibited item. can be specified.

(S306)
Next, when the examination is completed, that is, when the imaging is completed, the processes of steps S301 to S305 of recognizing the photographed X-ray image and presenting it to the inspector are completed, and when the examination is continuing, The processing from step S301 to step S305 is continued.

(S307)
Next, if you receive an instruction from the mouse or keyboard connected to the input I/F as to whether or not to generate or update a learning model using the X-ray images accumulated during operation, but do not generate or update the learning model. completes the entire process.

(S308)
Next, the article learning program 217 reads the X-ray image data 218, performs article learning, and registers the generated learning model in the learning model 219.

Similar to the step of recognizing objects, object learning uses libraries such as "Fully Convolutional Instance-aware Semantic Segmentation" and "Mask R-CNN," which are widely known as OSS (open source software) libraries. be able to.

Note that before learning about an article, contour information of the article and correct answer information for identifying the article are required, but in this embodiment, the method of assigning the correct answer is not limited. A general-purpose method may be used, such as manually enclosing the area of the item and selecting the item name.

Note that it is also possible to present the result recognized in step S303 as a candidate area of the article to improve the efficiency of the manual assignment of correct answers. For example, if an item can only be identified in one direction, the operator who assigns the correct answer can be shown the location where the same item may be shown in the other direction, thereby preventing omissions in assigning the correct answer. be able to. The learned learning model is registered in the learning model 219, and the process ends.

According to this embodiment, by integrating the recognition results of images taken in two directions, it is possible to improve the recognition accuracy of the article while presenting the image displayed to the inspector in a direction in which the article is easily seen. This has the effect of improving inspection efficiency while preventing inspection omissions. Further, even if the display is on one screen, necessary information can be displayed in an integrated manner, which has the effect of reducing the installation space of the display unit and the number of devices.

Although this embodiment shows an example in which images are taken mainly in two directions, it is sufficient if images are taken in a plurality of directions.
Further, in this embodiment, an example was shown in which the article is photographed using X-rays, but other electromagnetic waves such as terahertz waves may be used as long as they can penetrate the article and photograph the article.
For example, an electromagnetic wave image processing device and an electromagnetic wave image processing method that can improve the accuracy of recognizing prohibited items in an electromagnetic wave inspection device that can take images from multiple directions even with the following configuration in which X-rays are converted to electromagnetic waves. It has the same effect of being able to provide the following.
An electromagnetic wave image acquisition unit that acquires a plurality of electromagnetic wave images of an article taken from a plurality of directions, and an article recognition system that recognizes the article using a learning model that has learned the first side of the article from the plurality of electromagnetic wave images. An electromagnetic wave image processing device comprising: a recognition result integrating unit that integrates recognition results for a plurality of electromagnetic wave images; and a screen generation unit that generates screen information based on the integrated recognition results.
An electromagnetic wave image processing method in an electromagnetic wave image processing device, the method comprising an electromagnetic wave image acquisition step of acquiring a plurality of electromagnetic wave images taken from a plurality of directions of an article, and learning a first side of the article from the plurality of electromagnetic wave images. an article recognition step for recognizing an article using the learned learning model; a recognition result integration step for integrating recognition results for a plurality of electromagnetic wave images; and a screen generation step for generating screen information based on the integrated recognition results. An electromagnetic wave image processing method comprising:
Further, in this embodiment, an example is shown in which the X-ray apparatus main body 101 and the X-ray image processing apparatus 102 are separate bodies, but the X-ray image processing apparatus may be built in the X-ray apparatus main body.

Further, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to add, delete, or replace some of the configurations of the embodiments with other configurations.

100: X-ray inspection device 101: Device main body 102: X-ray image processing device 103: Display section 104: Input section 201: CPU
202: Main memory 205: Communication section 210: Storage section 211: OS
212: X-ray image acquisition program 213: Article recognition program 214: Recognition result integration program 215: Screen generation program 216: Calibration program 217: Article learning program 218: X-ray image data 219: Learning model

Claims (12)

an X-ray image acquisition unit that acquires a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition unit that recognizes the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration unit that integrates recognition results for the plurality of X-ray images;
a screen generation unit that generates screen information based on the integrated recognition results;
has
The recognition result integration unit selects an X-ray image with the highest reliability of the recognition result for the first surface of the article from among the plurality of X-ray images.
X-ray image processing device. The X-ray image processing device according to claim 1,
The article recognition unit recognizes an article using a learning model that has learned a first side of the article and a learning model that has learned a second side that is different from the first side of the article.
X-ray image processing device. an X-ray image acquisition unit that acquires a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition unit that recognizes the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration unit that integrates recognition results for the plurality of X-ray images;
a screen generation unit that generates screen information based on the integrated recognition results;
has
The article recognition unit recognizes an article using a learning model that has learned a first side of the article and a learning model that has learned a second side different from the first side of the article,
The recognition result integrating unit is capable of recognizing a first surface of the article in one of the plurality of X-ray images, and recognizing a second surface of the article in another one of the plurality of X-ray images. recognizing the article if it is recognizable;
X-ray image processing device. an X-ray image acquisition unit that acquires a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition unit that recognizes the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration unit that integrates recognition results for the plurality of X-ray images;
a screen generation unit that generates screen information based on the integrated recognition results;
has
When a dangerous object is recognized only in some of the X-ray images among the plurality of Add indications of areas that may contain objects,
X-ray image processing device. The X-ray image processing device according to claim 4,
The article recognition unit recognizes an article using a learning model that has learned a first side of the article and a learning model that has learned a second side that is different from the first side of the article.
X-ray image processing device. The X-ray image processing device according to any one of claims 1 to 5 ,
a calibration unit that adjusts the size or display position of the plurality of X-ray images;
X-ray image processing device. An X-ray image processing method in an X-ray image processing device, comprising:
an X-ray image acquisition step of acquiring a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition step of recognizing the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration step of integrating recognition results for the plurality of X-ray images;
a screen generation step for generating screen information based on the integrated recognition results;
has
The recognition result integration step selects an X-ray image with the highest reliability of the recognition result for the first surface of the article from the plurality of X-ray images.
X-ray image processing method. The X-ray image processing method according to claim 7 ,
The article recognition step recognizes the article using a learning model that has learned a first side of the article and a learning model that has learned a second side that is different from the first side of the article.
X-ray image processing method. An X-ray image processing method in an X-ray image processing device, comprising:
an X-ray image acquisition step of acquiring a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition step of recognizing the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration step of integrating recognition results for the plurality of X-ray images;
a screen generation step for generating screen information based on the integrated recognition results;
has
The article recognition step recognizes an article using a learning model that has learned a first side of the article and a learning model that has learned a second side that is different from the first side of the article,
In the recognition result integration step, a first surface of the article can be recognized in one of the plurality of X-ray images, and a second surface of the article can be recognized in another one of the plurality of X-ray images. recognizing the article if it is recognizable;
X-ray image processing method. An X-ray image processing method in an X-ray image processing device, comprising:
an X-ray image acquisition step of acquiring a plurality of X-ray images of the article taken from a plurality of directions;
an article recognition step of recognizing the article using a learning model that has learned a first surface of the article with respect to the plurality of X-ray images;
a recognition result integration step of integrating recognition results for the plurality of X-ray images;
a screen generation step for generating screen information based on the integrated recognition results;
has
In the screen generation step, when a dangerous object is recognized only in some of the X-ray images among the plurality of X-ray images, a dangerous object is added to a portion corresponding to the dangerous object in the remaining Add indications of areas that may contain objects,
X-ray image processing method. The X-ray image processing method according to claim 10,
The article recognition step recognizes the article using a learning model that has learned a first side of the article and a learning model that has learned a second side that is different from the first side of the article.
X-ray image processing method. The X-ray image processing method according to any one of claims 7 to 11 ,
a calibration step of adjusting the size or display position of the plurality of X-ray images;
X-ray image processing method.

Images