JP2020095707A - Alignment-free video change detection using deep blind image region prediction - Google Patents

Abstract

To provide a method of detecting a change in a scene between images captured at different times by a moving camera.SOLUTION: A method 400 of detecting a change in a scene between images captured at different times by a moving camera comprises generating 450 an image corresponding to a query image captured by the moving camera using a reconstruction model based on reference images captured by the moving camera, and detecting 460 a change in the scene by comparing the query image with the generated image.SELECTED DRAWING: Figure 4

Description

The present specification relates generally to image processing, and more particularly to detecting scene changes by comparing image sequences of the scene captured by each moving camera. The present specification also relates to a computer program product including a computer readable medium having a computer program recorded thereon for detecting a change in a scene by comparing image sequences of the scene captured from a mobile camera.

Public places such as shopping centers, parking lots, and railway stations are increasingly being monitored using large networks of video cameras. Video surveillance application areas include security, safety, traffic management, and business analytics. The large amount of video produced by camera networks requires an automated method known as "video analysis" to identify objects and events of interest and direct them to the user's attention. A basic task in video analysis, known as "change detection", is to determine which part of a scene has changed over time. Additional analysis, such as tracking and classification, is typically applied to the portion of the scene that is determined to have changed.

Methods for change detection in fixed cameras are available. In one method, called "background subtraction", a background model of the scene seen by a fixed camera is formed by estimating the distribution of pixel values at each image location according to a mixture of Gaussian models. Given a test frame, each pixel is classified as "background" if the pixel value has a high likelihood according to the background model, otherwise the pixel is labeled as "foreground". Foreground pixels are clustered into connected areas to detect moving objects. The drawback of this method is that the camera must remain fixed, otherwise the distributions at different pixel locations are mixed together and cannot be used to separate the foreground from the background.

Due to video surveillance over large geographical areas, fixed video camera networks are not practical, especially in remote areas where power and network infrastructure are limited. Applications of remote video surveillance include agricultural surveillance, critical infrastructure surveillance, border protection, exploration and rescue. In one method, surveillance cameras are attached to an airborne drone, which makes periodic patrols of railroad tracks along a fixed path. As mentioned above, conventional background subtraction methods cannot be used to detect scene changes in this scenario because the camera is constantly moving.

One method for detecting changes from a moving camera is, for example, by comparing two image sequences of a scene captured by the moving camera at different times during different patrols of the drone along the same fixed route. is there. The problem of comparing two image sequences of a scene to detect changes is referred to as "video change detection" throughout this disclosure. Further, the first image sequence is referred to as the "reference" sequence and the second image sequence is referred to as the "query" sequence throughout this disclosure.

Video change detection is difficult for several reasons. First, each frame from the moving camera is capturing a different portion of the entire scene so that a given frame from the reference sequence corresponds to the same portion of the scene as the given frame from the query sequence. There are times when you don't. Second, the trajectory of the camera, and thus the viewpoint of the camera, can change between the two videos due to localization errors and environmental conditions. A change in viewpoint, known as a “parallax error”, shifts part of the scene relative to other parts, which can be falsely detected as a change in the scene. Finally, the reference and query sequences are imaged in different environmental conditions, producing changes in observed brightness, shadows, and reflections, which can also be falsely detected as scene changes. ..

The above issues may be addressed by determining the temporal and spatial alignment of reference and query sequences. In one method, temporal alignment is determined by computing a low-dimensional feature vector for each frame and using an efficient search algorithm to find the nearest reference frame for each query frame in feature space. Given a pair of nearest neighbor frames, spatial registration is performed by determining pixel correspondences based on estimating the local homography between image patches. Finally, the scene change is detected by calculating the difference between the registered pixels. The drawback of this method is that scene changes cannot be distinguished from changes due to parallax error caused by viewpoint changes between the reference and query videos. Another drawback of this method is that the computational cost of temporal alignment increases proportionally with the length of the reference sequence. This can be a significant cost for large-scale practical applications such as perimeter protection or gas pipeline monitoring. Yet another drawback of this method is the inability to distinguish scene changes from illumination, shadow, and reflection changes that occur when reference and query sequences are imaged in different weather conditions.

Alternatively, the temporal and spatial alignment track a given region of interest (ROI) in both the reference and query videos and identify the keyframes corresponding to a particular viewpoint with the Global Positioning System (GPS) location. Established by associating with a viewpoint identifier such as. The keyframes from the reference sequence are used to learn the low dimensional eigenspace. Keyframes from the reference and query sequences with corresponding view identifiers are projected into eigenspace and the distance between the projected points is calculated. If the distance is greater than the threshold, the scene is determined to have changed. The drawback of this method is that it requires a reference sequence and a query sequence to be imaged from substantially the same viewpoint. In practice, this may be difficult to achieve as localization methods like GPS have errors of up to a few meters. Another drawback is that the comparison is performed in the eigenspace rather than the original image and does not localize certain changes in the ROI. Yet another drawback is that a given ROI needs to be tracked in both the reference and query sequences. No changes outside the ROI are detected.

It is an object of the present invention to substantially overcome, or at least ameliorate, one or more drawbacks of existing configurations.

The above problem by reconstructing a query image using an image patch predictor trained on a reference video sequence and detecting scene changes by comparing the reconstructed query image with the original query image. A configuration called Alignment Free Video Change Detection (AVCD) configuration is disclosed which seeks to address

According to one aspect of the disclosure, a method of detecting scene changes between images capturing a scene at different times, the images including a reference image and a query image, the method comprising: Reconstructing a query image using a. and a reconstruction model based on a reference image and detecting a scene change by comparing the query image with the reconstructed query image. A method is provided.

One or more embodiments of the present invention will be described with reference to the following drawings.
, 1A and 1B collectively show an example of a mobile camera that captures two image sequences of a scene at different times, where the AVCD configuration may be applied. , 2A and 2B are schematic block diagrams of a general-purpose computer system capable of implementing the described AVCD configuration. , , , 3A, 3B, 3C, and 3D collectively show an example of detecting scene changes according to one AVCD configuration. FIG. 4 is a schematic flow diagram showing a method for detecting a scene change according to one AVCD configuration. FIG. 5 is a schematic flow diagram showing a sub-process of training a reconstruction model based on a set of reference images as used in the method of FIG. FIG. 6 shows an example of forming an image patch around a keypoint according to the sub-process of FIG. FIG. 7 is a schematic flow diagram showing a sub-process of reconstructing a query image based on a trained reconstruction model as used in the method of FIG. FIG. 8 shows an example of selecting predicted pixel values at pixel locations as used in the sub-process of FIG. FIG. 9 is a schematic flow diagram showing a sub-process of predicting patches as used in the sub-processes of FIGS. 5 and 7. FIG. 10 shows an example of forming a donut shaped area around an image patch as used in the method of FIG.

When referring to steps and/or features having the same reference numeral in any one or more of the accompanying drawings, those steps and/or features are shown for the purpose of this description in opposite sense. Unless otherwise, they have the same function or behavior.

It should be noted that the "Background" section and the discussion contained in the section above relating to prior art arrangements relate to the discussion of documents or devices forming public knowledge through their respective publications and/or uses. Such discussions should not be construed as an expression by the inventor or patent applicant, and such documents or devices in any way form part of the general general knowledge in the art. ..

Context An image, such as image 310 in FIG. 3, is composed of visual elements. The terms “pixel”, “pixel position”, and “image position” are used interchangeably throughout this specification to refer to one of the visual elements in a captured image. Each pixel of an image is described by one or more values collectively referred to as a "pixel value" and characterizes the characteristics of the captured scene within the image. Pixel values include single intensity values (characterizing the brightness of the scene at pixel locations), triplets of values (characterizing the color of the scene at pixel locations), etc.

A “patch”, “image patch”, or “region” in an image, such as patch 820 in FIG. 8, refers to a collection of one or more spatially adjacent visual elements. A "keypoint" in an image is a local image structure that has a well-defined position and can be detected with high reproducibility despite local perturbations such as intensity changes or geometric deformations. An example of a key point is a "corner", which is a local image structure characterized by image gradients in multiple directions. Another example of a keypoint is a "blob," which is a local image structure characterized by high contrast between the central and surrounding areas. “Bounding box” refers to a straight boundary that surrounds a patch, region, keypoint, or object in an image, such as bounding box 322 in FIG. 3B. A "feature" or "image feature" represents a derived value or set of derived values determined from pixel values in a patch. Examples of features or image features include a histogram of color values within a patch, a histogram of quantized image gradient responses within a patch, a set of activations at a particular layer of an artificial neural network applied to a patch, and so on.

The present disclosure provides a method of comparing a reference image sequence and a query image sequence captured at different times to determine a scene change. 1A and 1B show exemplary use cases in which an AVCD configuration may be applied. The overall goal is to monitor the area near the gas pipeline 110 for changes due to failures, environmental threats, security threats, or other factors that may interfere with its operation. In this example, the query video and the reference video are captured by a video camera 135 attached to the airborne drone 130 as shown in FIG. 1A. In one configuration, the drone 130 is equipped with a computer system to which the AVCD configuration may be applied. In another configuration, the video captured by the camera 135 is transferred wirelessly or downloaded to a remote computer system where the AVCD configuration may be applied.

To capture the reference video, the drone 130 is deployed near the pipeline 110 and navigates along a predetermined path 120 while recording video in a region 140 near the pipeline 110. The query video is then captured by deploying the drone 130 to follow the path 125 (similar to the reference path 120) while recording the video in the area 140 near the pipeline 110. In practice, path 120 and path 125 may be the same even if the drone navigates using the same waypoint during both deployments due to inaccuracies in location and flight control. Is low. Therefore, the reference and query videos capture the scene from somewhat different perspectives.

In the example shown in FIG. 1B, vehicle 150 enters region 140 near pipeline 110 during the time after capturing the reference video and before capturing the query video. The computer system compares the reference video with the query video, detects changes in the scene due to the presence of a vehicle, and triggers an appropriate response. Examples of suitable responses include computer systems that apply additional analysis to query and reference videos to classify scene changes, computer systems that send information about detected changes to a user, and the like.

This exemplary AVCD configuration applies to a range of applications both inside and outside the field of video surveillance. In one application, cameras are mounted on ground vehicles such as trucks or trains and used to monitor defects in transportation infrastructure such as roads or train lines. In another application, the reference sequence includes an image of the patient's healthy internal tissue imaged using a medical imaging method such as a CT scan, and the query image represents the same internal tissue imaged later. The AVCD configuration is applied to detect tissue changes that indicate a health risk. In yet another application, the reference sequence comprises an image of a correctly manufactured batch of integrated circuits taken during a particular point in the manufacturing process. The AVCD construction is applied to the query images in later batches to detect manufacturing defects.

Overview Capturing a reference image sequence and a query image; training a reconstruction model based on the reference image; reconstructing a query image using the trained reconstruction model; detecting scene changes Comparing the query image with the reconstructed query image to do so. The disclosed AVCD configuration does not require reference image sequences and query images to image a scene from the same viewpoint or under the same environmental conditions. Moreover, the disclosed AVCD configuration does not require temporal or spatial alignment of the query image with respect to the reference image. This is because the AVCD configuration compares the query image with the reconstructed query image that is already aligned with each other and has the same viewpoint and environmental conditions. Finally, the disclosed AVCD construction can be implemented with fixed computational cost for the comparison and detection steps, independent of the length of the reference sequence.

3A, 3B, 3C, and 3D collectively show an example of comparing a reference image sequence and a query image to detect a scene change according to one AVCD configuration. The reference image is used collectively to train the reconstruction model.

FIG. 3A shows an example of a single image frame 310 of a reference image sequence, the single image frame 310 including a patch 312. In one AVCD configuration, the reconstruction model includes predicting pixel values in patch 312 based on features extracted from pixels in annular region 316. The annular region 316 is bounded by the boundary of the patch 312 and the boundary of the larger concentric patch 315. Examples of the patch prediction method include a dictionary learning model, an artificial neural network (also called a deep neural network), and the like. In one AVCD configuration, the reconstruction model is trained using multiple patches extracted from multiple reference images.

FIG. 3B shows an example of the query image 320 and shows the same scene portion as the reference image 310 of FIG. 3A. During the time between capturing the reference image 310 and capturing the query image 320, the scene changed due to the advent of the vehicle 327.

FIG. 3C shows an example of a reconstructed query image 330 calculated by processing the query image 320 using a reconstructed reconstruction model for reference images including the reference image 310 of FIG. 3A. The pixel value 333 in the reconstructed query image 330 is partially determined by predicting the patch 332 that contains the pixel 333. The patch 332 is predicted based on the features extracted from the pixels in the annular region between the corresponding co-located patch 322 in the query image 320 and the larger concentric patch 325 in the query image 320. .. The reconstruction process is applied to all pixel locations to determine the entire reconstructed query image 330. Vehicles 327 in query image 320 do not appear in reconstructed query image 330 because vehicles 327 cannot be predicted by the reconstructed model trained on reference image 310 in which vehicle 327 does not appear.

FIG. 3D shows an example of a change mask 340 determined by comparing the reconstructed query image 330 of FIG. 3C with the query image 320 of FIG. 3B. In one AVCD configuration, the value at pixel position 343 in change mask 340 is between the value at corresponding pixel position 333 in reconstructed query image 330 and the corresponding pixel position 323 in query image 320. Is a binary value determined by calculating the absolute difference of and applying a threshold.

In the example of FIG. 3D, 333 pixels of the reconstructed query image 330 do not match 323 pixels of the query image 320, so the pixel value at position 343 of the change mask is assigned a binary value of 1. Conversely, the 334 pixels of the reconstructed query image 330 match the 324 pixels of the query image 320, so the 344 pixels of the change mask 340 are assigned the binary value 0.

Structural Environment FIGS. 2A and 2B illustrate a general purpose computer system 250 on which the various AVCD configurations described may be implemented.

As seen in FIG. 2A, computer system 250 includes computer module 201, keyboard 202, mouse pointer device 203, one or more cameras such as scanner 226, cameras 261 and 262, and input such as microphone 280. Devices and output devices including printers 215, display devices 214, and speakers 217. An external modulator-demodulator (modem) transceiver device 216 may be used by computer module 201 to communicate with a remote camera, such as 260, via communication network 220 via connection 221. Communication network 220 may be the Internet, a cellular telecommunications network, or a wide area network (WAN) such as a private WAN. If connection 221 is a telephone line, modem 216 can be a conventional "dial-up" modem. Alternatively, if connection 221 is a high capacity (eg, cable) connection, modem 216 may be a broadband modem. A wireless modem may also be used for wireless connection to communication network 220.

Computer module 201 typically includes at least one processor unit 205 and memory unit 206. For example, the memory unit 206 may include a semiconductor RAM (random access memory) and a semiconductor ROM (read only memory). Computer module 201 includes audio-video interface 207 coupled to video display 214, speaker 217, and microphone 280, keyboard 202, mouse 203, scanner 226, camera 261, and optionally a joystick or other human interface device (not shown). It also includes a number of input/output (I/O) interfaces, including an I/O interface 213 coupled to the I/O interface 213, and an interface 208 for external modem 216 and printer 215. In some implementations, modem 216 may be incorporated within computer module 201, eg, interface 208. Computer module 201 also has a local network interface 211 that enables coupling of computer system 250 via a connection 223 to a local area communication network 222 known as a local area network (LAN). As shown in FIG. 2A, local communication network 222 may also be coupled to wide network 220 via connection 224, which typically includes so-called “firewall” devices or devices of similar function. The local network interface 211 may comprise an Ethernet circuit card, a Bluetooth wireless configuration, or an IEEE 802.11 wireless configuration, but the interface 211 may implement many other types of interfaces. You can

The I/O interfaces 208 and 213 can provide either or both serial and parallel connectivity, the former typically implemented according to the Universal Serial Bus (USB) standard and corresponding USB connectors (see FIG. (Not shown). A storage device 209 is provided and typically includes a hard disk drive (HDD) 210. Other storage devices such as floppy disk drives and magnetic tape drives (not shown) can also be used. Optical disc drive 212 is typically provided to act as a non-volatile source of data. Portable memory devices such as, for example, optical discs (eg, CD-ROM, DVD, Blu ray DiscTM), USB-RAM, portable, external hard drives, and floppy discs are suitable sources of data for system 250. Can be used as

The components 205-213 of the computer module 201 typically communicate via an interconnected bus 204 to effect conventional operating modes of the computer system 250 known to those skilled in the art. For example, processor 205 is coupled to system bus 204 using connection 218. Similarly, memory 206 and optical disc drive 212 are coupled to system bus 204 by connection 219. Examples of computers that can implement the described arrangements include IBM-PC and compatibles, Sun Sparcstations, Apple MacTM, or similar computer systems.

The AVCD method can be implemented using a computer system 250, and the processes of FIGS. 4, 5, 7, and 9 described include one or more AVCD software application programs executable within the computer system 250. 233 can be implemented. Specifically, the steps of the AVCD method are performed by instructions 231 (FIG. 2B) in software 233 executed in computer system 250. Software instructions 231 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, the first part and the corresponding code module performing the AVCD method, and the second part and the corresponding code module of the first part and the user. Manage the user interface between.

The AVCD software may be stored on a computer readable medium, including the storage devices described below, for example. The software is loaded into computer system 250 from a computer-readable medium and then executed by computer system 250. A computer-readable medium having such software or computer program recorded on the computer-readable medium is a computer program product. The use of computer program products in computer system 250 preferably provides an advantageous apparatus for implementing the AVCD method.

The software 233 is typically stored in the HDD 210 or the memory 206. The software is loaded into computer system 250 from a computer readable medium and executed by computer system 250. Thus, for example, the software 233 may be stored on an optically readable disc storage medium (eg, CD-ROM) 225 that is read by the optical disc drive 212. A computer-readable medium having such software or computer program recorded on it is a computer program product. The use of computer program products in computer system 250 preferably results in an apparatus for implementing AVCD configuration.

In some cases, the AVCD application program 233 may be encoded on one or more CD-ROMs 225 and provided to the user and read via the corresponding drive 212, or read by the user from the network 220 or 222. May be Further, the software may be loaded into computer system 250 from other computer readable media. Computer-readable storage media refers to any non-transitory tangible storage media that provides recorded instructions and/or data to computer system 250 for execution and/or processing. Examples of such storage media include floppy disks, magnetic tapes, CD-ROMs, DVDs, Blu-ray TM disks, hard disk drives, ROMs or integrated circuits, USB memories, magneto-optical disks. , Or a computer readable card, such as a PCMCIA card, regardless of whether such a device is internal or external to computer module 201. Examples of temporary or non-tangible computer readable transmission media that may also be involved in providing software, application programs, instructions and/or data to computer module 201 include wireless or infrared transmission channels, as well as another computer or networked. Network connection to the device, as well as the Internet or Intranet, including information recorded on email transmissions and websites.

The second portion of application program 233 and the corresponding code module described above may be executed to implement one or more graphical user interfaces (GUIs) rendered or rendered on display 214. Through operation of the keyboard 202 and mouse 203, typically the user of the application and computer system 250 operates the interface in a functionally adaptable manner to provide control commands and/or inputs to the application associated with the GUI. can do. Other forms of functionally adaptable user interfaces may also be implemented, such as audio interfaces that utilize speech prompts output via speaker 217 and user voice commands input via microphone 280.

FIG. 2B is a detailed schematic block diagram of processor 205 and “memory” 234. The memory 234 represents a logical collection of all memory modules (including the HDD 209 and the semiconductor memory 206) accessible by the computer module 201 of FIG. 2A.

When the computer module 201 is first turned on, the power-on self-test (POST) program 250 is executed. The POST program 250 is typically stored in the ROM 249 of the semiconductor memory 206 of FIG. 2A. A hardware device, such as ROM 249, that stores software is sometimes referred to as firmware. The POST program 250 tests the hardware within the computer module 201 to ensure proper functionality and is typically stored in processor 205, memory 234 (209, 206), and also typically ROM 249. The basic input/output system software (BIOS) module 251 being tested is checked for correct operation. When the POST program 250 is normally executed, the BIOS 251 activates the hard disk drive 210 of FIG. 2A. When the hard disk drive 210 is activated, the bootstrap loader program 252 resident in the hard disk drive 210 is executed via the processor 205. As a result, the operating system 253 is loaded into the RAM memory 206, and the operating system 253 starts operating on it. Operating system 253 is a system level application executable by processor 205 to perform various high level functions, including processor management, memory management, device management, storage management, software application interfaces, and general user interfaces.

The operating system 253 manages the memory 234 (209, 206) to have sufficient memory for each process or application executing on the computer module 201 to execute without conflicting with memory allocated to another process. Guaranteed to have. Moreover, the different types of memory available in the system 250 of FIG. 2A must be used appropriately so that each process can execute effectively. Accordingly, aggregate memory 234 is not intended to indicate how a particular segment of memory is allocated (unless otherwise noted), and is a general view of memory accessible by computer system 250 and its It is intended to provide how such things are used.

As shown in FIG. 2B, the processor 205 includes a number of functional modules including a control unit 239, an arithmetic logic unit (ALU) 240, and local or internal memory 248, sometimes referred to as cache memory. Cache memory 248 typically includes a number of storage registers 244-246 in a register section. One or more internal buses 241 functionally interconnect these functional modules. Processor 205 also typically has one or more interfaces 242 for communicating with external devices via system bus 204 using connection 218. Memory 234 is coupled to bus 204 using connection 219.

AVCD application program 233 includes a series of instructions 231 that can include conditional branch and loop instructions. Program 233 may also include data 232 used to execute program 233. Instruction 231 and data 232 are stored in memory locations 228, 229, 230 and 235, 236, 237, respectively. Depending on the relative size of instruction 231 and memory locations 228-230, the particular instruction may be stored in a single memory location, as indicated by the instruction shown in memory location 230. Alternatively, the instructions may be segmented into a number of portions, each stored in a separate memory location, as indicated by the instruction segments shown in memory locations 228 and 229.

Generally, processor 205 is provided with a set of instructions to execute there. Processor 205 waits for the next input and processor 205 responds to this input by executing another set of instructions. Each input is data generated by one or more of the input devices 202, 203, data received from an external source via one of the networks 220, 202, of the storage devices 206, 209. It may be provided from one or more of several sources, including data retrieved from one or retrieved from a storage medium 225 inserted into a corresponding reader 212, all in FIG. 2A. It is shown. Execution of the set of instructions may result in the output of data in some cases. Execution may also include storing data or variables in memory 234.

The disclosed AVCD configuration uses input variables 254 stored in corresponding memory locations 255, 256, 257 in memory 234. The AVCD configuration produces a set of output variables 261, which are stored in memory 234 at corresponding memory locations 262, 263, 264. Intermediate variables 258 may be stored in memory locations 259, 260, 266, and 267.

Referring to the processor 205 of FIG. 2B, the registers 244, 245, 246, the arithmetic logic unit (ALU) 240, and the control unit 239 “fetch, decode, And execute" cycles to perform the sequence of micro-operations required to perform the cycle. Each fetch, decode, and run cycle is
A fetch operation for fetching or reading the instruction 231 from the memory locations 228, 229, 230 a control unit 239 decode operation for determining which instruction has been fetched a control unit 239 and/or an ALU 240 execute the instruction Have an action.

Thereafter, additional fetch, decode, and execute cycles for the next instruction may be performed. Similarly, a save cycle is performed in which control unit 239 saves or writes a value to memory location 232.

Each step or sub-process in the processes of FIGS. 4, 5, 7 and 8 is associated with one or more segments of program 233 and within register sections 244, 245, 247, ALU 240, and processor 205. Control unit 239 cooperates to execute by performing fetch, decode, and execute cycles for all instructions in the instruction set for the segment of interest of program 233.

The AVCD method may alternatively be implemented in dedicated hardware, such as one or more integrated circuits that perform AVCD functions or subfunctions. Such dedicated hardware may include a graphics processor, digital signal processor, or one or more microprocessors and associated memory, and may reside on a platform such as a video camera.

AVCD Method FIG. 4 shows a method 400 for detecting scene changes by comparing a reference image sequence and a query image according to one AVCD configuration. The method 400 may be implemented as one or more software code modules of a software application program 233, whose execution is controlled by the processor 205, provided in the hard disk drive 210. The following description provides details, examples, and alternative implementations for the major steps of method 400. Further details, examples, and alternative implementations of sub-processes 430 and 450 are described below with reference to FIGS. 5 and 7, respectively.

Method 400 begins with a first readout step 420, where a set of reference images is received as input. In one example, a reference image shows a scene captured by a mobile camera mounted on an unmanned aerial vehicle navigating along a predetermined path. In this example, when the camera is moving, the reference images capture the scene from different viewpoints. In another example, a reference image is imaged by the medical imaging device to show the healthy tissue of the patient.

Control then transfers from step 420 to a training sub-process 430 which trains the reconstruction model based on the reference image. The reconstruction model is then used to determine the reconstructed query image of the query image such that the reconstructed query image includes aspects of scene structure in the query image that were present in the reference image. You can Aspects of the scene that were not present in the reference image are not displayed in the reconstructed query image. Therefore, the function of the training process is to learn in part the scene structure represented in the reference image. Further details, examples, and alternative implementations of sub-process 430 are described below with reference to FIG.

Control then transfers from sub-process 430 to a second receiving step 440, which receives a query image corresponding to the same scene as the reference image. The query image is captured at a different time than the reference image. In one AVCD configuration, the query image captures the same scene as the reference image from a somewhat different perspective due to the imaging process. In one example, the query image shows a scene imaged by a mobile camera onboard the drone navigating along a predetermined path similar to the path previously used to image the reference image sequence. In another example, the query image shows patient tissue previously imaged in the reference image using the same medical imaging device.

In one configuration, steps 420 and 440 may be performed before sub-process 430. Therefore, in this configuration, steps 420 and 440 are performed before subprocess 430. Upon completion of sub-process 430, method 400 in this configuration proceeds to step 450.

Control then transfers from step 440 to a reconstruction sub-process 450 that determines a reconstructed query image based on the query image received in step 440 and the reconstruction model trained on the reference image in sub-process 430. .. The reconstructed query image includes aspects of the scene structure in the query image that were present in the reference image. The aspect of the scene that changed between the capturing of the reference image and the query image is not reconstructed. Further details, examples, and alternative implementations of step 450 are described below with reference to FIG.

Control then transfers from sub-process 450 to detection step 460 which determines a scene change based on the query image received in step 440 and the reconstructed query image determined in sub-process 450. In one AVCD configuration, the detection step 460 includes querying the corresponding reconstructed query image 330 determined in sub-process 450, as shown in FIGS. 3B, 3C and 3D. The change mask 340 is determined by comparing with the image 320. The value at pixel position 343 in change mask 340 calculates the absolute difference between the value at corresponding pixel position 333 in reconstructed query image 330 and the corresponding pixel position 323 in query image 320. , A binary value determined by applying a threshold. An example of applying a threshold is to assign a binary value of 1 if the difference is greater than 30, and a value of 0 otherwise. Changes in the scene are determined to have occurred at pixel locations in the change mask that contain non-zero values.

Embodiments of step 460 that include post-processing the change mask before determining the scene change may be performed at step 460 as well. In one alternative AVCD configuration, a set of morphological filtering operations are applied to the binary change mask to remove noise. One example of a morphological filtering operation is binary erosion. Another example of a morphological filter operation is a binary extension. In another alternative AVCD configuration, connected component analysis is applied to the varying mask to identify distinct regions. In yet another alternative AVCD configuration, the area of each detected object is determined and any region with an area below a fixed area threshold is discarded by setting the corresponding pixel value in the binary mask to zero. An example of a fixed area threshold is a 10x10 square pixel. In yet another alternative AVCD configuration, additional features of the detected area are calculated. One example of an additional feature is the center of gravity. Another example of an additional feature is a bounding box. Yet another example of an additional feature is the second moment of area. The rules are applied to the additional features to determine if the corresponding area should be discarded. One example of a rule is to discard an area if the centroid lies outside the area of interest (AOI) in the image. One example of an AOI is a bounding box that indicates a region of the image that the user has specified in the image and the user wants to detect changes.

Method 400 ends after completing detection step 460.

FIG. 5 shows a sub-process 430 for training a reconstruction model based on a sequence of reference images. Sub-process 430 will now be described using the example embodiment from FIG. Sub-process 430 may be implemented as one or more software code modules of software application program 233, which is provided on hard disk drive 210 and is controlled in its execution by processor 205. The following description provides details, examples, and alternative implementations of the main steps of sub-process 430.

Sub-process 430 begins with a detection step 530 which detects keypoints for each of the reference images received in step 420. A "keypoint" in an image is a local image structure that has a well-defined position and can be detected with high reproducibility despite local perturbations such as intensity changes or geometric deformations. An example of a key point is a "corner", which is a local image structure characterized by image gradients in multiple directions. Another example of a keypoint is a "blob," which is a local image structure characterized by high contrast between the central and surrounding areas. The key points can be detected using a corner detection method, a blob detection method, or the like. Examples of corner detection methods include Harris corner detectors, FAST corner detectors, Shi-Tomasi corner detectors, and the like. Examples of blob detection methods include difference of Gaussians (DOG) blob detectors, maximally stable extremal regions (MSER) detectors, and the like.

In one AVCD configuration, the keypoint detector applied at step 530 produces a response value associated with each detected keypoint. The response value indicates the strength of the response of the keypoint to the keypoint detector. The keypoint detector is applied to the entire image and keypoints with a response value below a fixed threshold are discarded. An example of a fixed threshold is 0.001 for a DOG corner detector.

In another AVCD configuration, the image is divided into a grid of fixed size non-overlapping cells, the detected keypoint with the highest response value in each grid cell is retained, and all other keypoints are discarded. .. An example of a fixed size is 16x16 pixels.

Control then proceeds from step 530 to decision step 535 which determines one or more patch bounding boxes in each of the reference images. In one AVCD configuration, the patch bounding box is determined for each keypoint detected in step 530. With reference to the exemplary embodiment of FIG. 6, the patch bounding box 620 in the reference image 600 is determined as a fixed size square centered on the keypoint 610. Therefore, the above arrangement determines the patch bounding box based on the detected keypoints.

In another AVCD configuration, the patch bounding box is determined at every pixel position in the reference image as a fixed size rectangle centered at the pixel position. In yet another AVCD configuration, in step 535, a set of non-overlapping patch bounding boxes is determined by dividing the reference image into a regular grid of fixed size square patches. An example of a fixed size is 16x16 pixels.

Control then transfers from step 535 to an initialization step 540 which initializes the reconstruction model. In one AVCD configuration, the reconstruction model is a convolutional deep neural network and the weights in the convolutional deep neural network are randomly initialized. In another AVCD configuration, the weights in the deep neural network are partially initialized by retrieving the weights from the deep neural network previously trained in different tasks. An example of a different task is image classification. Those skilled in the art will recognize that other methods for initializing weights in deep neural networks may be performed in step 540 as well.

Control then transfers from step 540 to a prediction sub-process 550 that calculates a predicted patch for each patch determined in step 535 using the reconstruction model. As a result, the reconstructed model used in the first execution of step 550 is the model initialized in step 540. The reconstruction model used in subsequent executions of step 550 is the model updated in step 570 (discussed below).

In one AVCD configuration, the predictive image patch processes the reference image using a convolutional deep neural network (ie, a reconstruction model) and outputs the output layer of the convolutional deep neural network corresponding to the patch bounding box determined in step 535. Determined by selecting the activation from. Further details, examples, and alternative implementations of sub-process 550 are described below with reference to method 900 of FIG.

Control then transfers from sub-process 550 to calculation step 560 which calculates training loss based on the patch determined in step 535 and the predicted patch determined in sub-process 550. In one AVCD configuration, the training loss, known as the squared error loss, is the same as the pixel in the patch from the reference image (determined in step 535) and the corresponding predicted patch (calculated in subprocess 550). It is calculated as the sum of the squared differences between the pixels in position over all pixels in all patch bounding boxes.

In another AVCD configuration, the training loss, known as the binary cross-entropy loss, is the product of all the pixel values in the predicted patch and the logarithm of the pixel values at the same position in the corresponding patch from the reference image. Computed as the negative number of the sum over all pixels in the patch bounding box of. Those skilled in the art will appreciate that other training losses can be calculated in step 560 as well.

Control then transfers from step 560 to an update step 570 where the reconstruction model is updated based on the training loss determined in step 560. In one AVCD configuration, the reconstruction model is a deep neural network and the weights in the model are updated by applying one iteration of the iterative optimization algorithm. One example of an iterative optimization algorithm is the stochastic gradient descent method based on the gradient of the loss function determined using backpropagation. Another example of an iterative optimization algorithm is AdaGrad. One of ordinary skill in the art will recognize that one iteration of another iterative optimization algorithm may be similarly calculated at step 560.

Control then passes from step 570 to decision step 580 which determines if the training has converged and should be terminated. In one AVCD configuration, training is determined to have converged if the number of training iterations, ie, the number of iterations of steps 550, 560, and 570, exceeds a fixed iteration threshold. An example of a fixed repeat threshold is 250 repeats.

In another AVCD configuration, training is determined to have converged if the training loss determined in step 560 is below the fixed loss threshold. An example of a fixed loss threshold for binary cross entropy loss is 0.6. One of ordinary skill in the art will recognize that other convergence criteria may be applied in step 570 as well.

If it is determined that the training has not converged (NO), then control transfers from step 580 to the predictive subprocess 550. If it is determined that the training has converged (YES), then sub-process 430 ends.

Those skilled in the art will recognize that variations of sub-process 430 for training the reconstruction model may be implemented as well. In an alternative embodiment of sub-process 430, the training data is augmented by creating additional reference images based on the one or more randomized transforms of the reference images received in step 420. Examples of randomization transforms include random rotation of the reference image, random scaling of the reference image, translation of the reference image, random gamma correction applied to the reference image, and so on. Those skilled in the art will recognize that other randomizing transforms can be similarly applied to generate the expanded training data.

In another alternative embodiment of sub-process 430, step 540 initializes multiple instances of the reconstruction model with different random initializations. The training process of steps 550, 560, 570 and 580 is applied independently to each of the multiple instances of the reconstruction model. Finally, an additional selection step (not shown in Figure 5) selects the instance of the reconstruction model with the lowest training loss and discards the other instances of the reconstruction model.

FIG. 7 illustrates a sub-process 450 for reconstructing a query image based on the reconstruction model trained using the sub-process 430 of FIG. Sub-process 450 will now be described using the illustrative examples from FIGS. 6 and 8. Sub-process 450 may be implemented as one or more software code modules of software application program 233, which resides on hard disk drive 210 and whose execution is controlled by processor 205. The following description provides details, examples, and alternative implementations of the main steps of method 700.

Sub-process 450 begins at read step 720, where a query image (received at step 440 of method 400) and a reconstruction model (trained using sub-process 430 of FIG. 5) are received as inputs. ..

Control then transfers from step 720 to decision step 725 which determines the set of patch bounding boxes on the query image received in step 720. The set of patch bounding boxes includes one or more patch bounding boxes. In one AVCD configuration, the patch bounding box is determined at every pixel location in the query image. The patch bounding box is determined in the same way as shown in FIG. In FIG. 6, the patch bounding box 620 at the pixel position 610 of the reference image 600 is determined as a fixed size square boundary centered at the pixel position 610. An example of a fixed size is 16x16 pixels.

Control then proceeds from step 725 to a detection sub-process that computes a predicted patch corresponding to each patch bounding box determined in step 725 based on the reconstruction model trained using sub-process 430 of FIG. Move to 730. Sub-process 730 shares an embodiment with sub-process 550 of sub-process 430 of FIG. Further details, examples, and alternative implementations of sub-process 730 are described below with reference to method 900 of FIG.

Control then transfers from step 730 to calculation step 740 which calculates a prediction error for each predicted patch determined in step 730. In one AVCD configuration, the prediction error is calculated as the distance between the patch in the query image received at step 720 and the corresponding predicted patch determined at step 730. In one AVCD configuration, the calculated distance is the squared sum of the differences between the corresponding pixels in the pair of patches from steps 725 and 730. In another AVCD configuration, the calculated distance is the normalized cross-correlation between the pair of patches from steps 725 and 730. One of ordinary skill in the art will appreciate that other distance measurements may be similarly calculated at step 740.

Control then transfers from step 740 to selection step 750 which reconstructs the query image by selecting pixel values at each pixel location based on the patch predicted in step 730 and the prediction error calculated in step 740. .. FIG. 8 illustrates selecting a pixel value at position 810 in reconstructed query image 800. In one AVCD configuration, pixel values are selected based on a single patch prediction. In one example, the center pixel of the predicted patch 830 is selected and stored at position 810 in the reconstructed query image 800.

In another AVCD configuration, pixel values are selected based on multiple predictions. First, the set of all predicted patches, including patches 820 and 830, that include the pixel location 810 within the bounding box is determined. Next, the patch with the smallest prediction error determined in step 740 is selected from this set. Finally, the pixel value predicted at location 810 by the selected patch is stored at location 810 in the reconstructed query image.

Those skilled in the art will appreciate that variations of the embodiments described above for sub-process 450 may be similarly implemented. In one alternative AVCD configuration, multiple patches containing pixels 810 are selected in step 750 based on selection criteria. One example of a selection criterion is that the corresponding prediction error determined in step 740 is below a fixed threshold. An example of a fixed threshold is 0.1. With this configuration, the functionality of the selected patch is determined and stored at location 810 in the reconstructed query image. One example of a function is the average predicted value at location 810. Another example of a function is median prediction at location 810. In another alternative AVCD configuration, the set of non-overlapping patch bounding boxes is determined in step 725 by dividing the query image into a regular grid of fixed-size square patches. In this configuration, at step 730, each pixel location is predicted by a single patch in the grid. At step 750, the pixel value at each position in the reconstructed query image is obtained from the corresponding predicted patch.

Sub-process 450 ends after completing selection step 750.

FIG. 9 illustrates a method 900 for predicting patches based on a reconstructed model trained using the sub-process 430 of FIG. Method 900 is used by sub-process 550 shown in FIG. 5 and sub-process 730 shown in FIG. Method 900 will now be described using the illustrative example from FIG. Method 900 may be implemented as one or more software code modules of software application program 233, which resides on hard disk drive 210 and whose execution is controlled by processor 205.

The method 900 begins at a read step 920 and is trained using an image (either a reference image for sub-process 430 or a query image for sub-process 450), sub-process 430 of FIG. 5 (or The reconstructed model (initialized) and the set of patch bounding boxes (see steps 535 and 725) are received as inputs.

Then, control transfers from step 920 to preprocessing step 930 which applies preprocessing to the received image. In one AVCD configuration, pixel values in the received image are converted to a specific color space such as intensity color space, CIELAB color space. In another AVCD configuration, the received image is cropped and resized to a fixed size, such as 256x256 pixels. In yet another AVCD configuration, the image is filtered using a noise reduction filter, such as a Gaussian blur filter with a standard deviation of 2 pixels. In yet another AVCD configuration, preprocessing step 930 does not perform image preprocessing.

Control then proceeds from step 930 to an extraction step of extracting features from the annular region surrounding each patch based on the patch bounding box and reconstruction model received at step 920 and the preprocessed image determined at step 930. 940. The annular area surrounding each patch may be defined by the area 316 between patches 312 and 315, as shown in FIG. 3A.

In one AVCD configuration, feature extraction is implemented as a convolutional deep neural network with one or more layers of convolution filters that are ring-shaped. FIG. 10 shows an example of a square convolution filter 1030 and a circular convolution filter 1010 applied to the image 1000. The shape of square convolution filter 1030 is defined by bounding box 1035, which includes only non-zero weights of shaded regions within bounding box 1035. The shape of the annular convolution filter 1010 is defined by the outer bounding box 1005 and the inner bounding box 1020. The circular convolution filter 1010 includes only non-zero weights in the shaded area between the outer bounding box 1005 and the inner bounding box 1020.

In one example of a convolutional deep neural network used to extract features, the input layer is a single-channel grayscale image. The next layer has 64 square convolution filters of size 3x3x1 applied with a stride of 1 and normalized linear unit (ReLU) activation. The next layer has 64 square convolution filters of size 3x3x64 applied with 1 kernel extension, 1 stride, and ReLU activation. The next layer applies max pooling with a 2x2 size pooling kernel and a stride of 1. The next layer has 128 square convolution filters of size 3x3x64 applied with a kernel extension of 2, a stride of 1, and ReLU activation. The next layer applies maximum pooling with a pooling kernel of size 2x2, which is applied with a kernel extension of 2 and a stride of 1. The next layer has 192 square convolution filters of size 3x3x128 applied with 4 kernel extensions, 1 stride, and ReLU activation. The next layer applies max pooling with a pooling kernel of size 2x2, which is applied with a kernel extension of 4 and a stride of 1. The next layer has 384 square convolution filters of size 3x3x192 applied with 8 kernel extensions, 1 stride, and ReLU activation. The next layer applies max pooling with a 2x2 size pooling kernel, which is applied with a kernel extension of 8 and a stride of 1. The next layer has 768 ring-shaped convolution filters with an outer size of 7×7 and an inner size of 3×3, applied with 16 kernel extensions, 1 stride, and ReLU activation. .. The feature extracted in step 940 of method 900 is the activation of the annular filter. The convolution filter at all layers is applied with zero padding so that the features are determined for all positions in the image received in step 920.

Those skilled in the art will appreciate that variations of the embodiments of the convolutional deep neural network described above can be implemented as well. In an alternative embodiment, batch normalization is applied between each convolutional layer. In another alternative embodiment, ReLU activation is replaced with a different activation function. One example of an activation function is linear activation. Another example of an activation function is Sigmoid activation. In another alternative embodiment, the maximum pooling layer is replaced with a different pooling function. One example of a pooling function is average pooling. In another alternative embodiment, an annular filter is defined by the outer bounding box and has a weight that gradually decreases to zero at the center of the filter. In one example, the filter weights are the product of the learned weights and the two-dimensional inverse Gaussian function centered on the bounding box. The inverse Gaussian function is defined as 1 minus the Gaussian function with a fixed standard deviation. An example of a fixed standard deviation is one tenth of the width of the outer bounding box.

Control then transfers from step 940 to a prediction step 950 which calculates a predicted image patch corresponding to each patch bounding box received in step 920 based on the features extracted in step 940. In one AVCD configuration, the reconstruction model is a convolutional deep neural network and the prediction step is performed using a network layer applied to the output of the feature extraction network. In one example, the prediction network comprises 256 layers of convolutional filters of size 1×1×768. The output of the convolutional filter is mapped into a bounded interval using a bounded activation function. One example of a bounded activation function is sigmoid activation that limits the output to the range [0,1]. Finally, the predicted patch is formed by reshaping the output activation according to the size of the patch received in step 920. In one example, the 256 output activations at a particular location are reshaped into a 16x16 patch corresponding to the expected patch at that location. Those skilled in the art will recognize that other fixed filter sizes and activation functions may be used in the convolutional deep neural network as well, at step 950.

Method 900 ends after completing prediction step 950.

The above description of sub-processes 430 and 450, and method 900 provide examples based on a convolutional deep neural network reconstruction model to embodiments. Those skilled in the art will recognize that in alternative embodiments of sub-processes 430 and 450 and method 900, other machine learning models may be used as the reconstruction model as well. In one alternative AVCD configuration, the reconstruction model is based on dictionary learning using a dictionary with a fixed number of atoms. An example of a fixed number is 1024 atoms. In one embodiment of step 540 of sub-process 430, the dictionary atoms are randomly initialized. In another embodiment of step 540 of sub-process 430, the dictionary atom is initialized by applying K-singular value decomposition (K-SVD). In one embodiment of step 570 of sub-process 430, the dictionary is updated by applying one iteration of the alternating minimization algorithm. In one embodiment of step 940 of method 900, fixed dimensional features are calculated by vectorizing the pixel values in the patch bounding box received in step 920. An example of a fixed dimension is 256, which corresponds to vectorization of grayscale pixel values within a 16x16 pixel bounding box. In one embodiment of step 950 of method 900, a dictionary encoding is calculated based on the features calculated in step 940 and the dictionary atom received in step 920, and then the product of the coding coefficient and the corresponding dictionary atom. The patch is predicted by reconstructing the patch by computing the sum over. In one example, the dictionary encoding is determined using the least angle regression method.

Claims (14)

A method of detecting a scene change between images captured at different times by a moving camera, comprising:
Generating an image corresponding to the query image captured by the mobile camera using a reconstruction model based on a reference image captured by the mobile camera;
Detecting changes in the scene by comparing the query image and the generated image;
A method comprising: Furthermore,
Training the reconstruction model from the reference image, the step comprising:
Determining one or more patch bounding boxes in the reference image;
Calculating a predictive patch for the one or more patch bounding boxes in the reference image using the reconstruction model;
Calculating a training loss based on the one or more patch bounding boxes of the reference image and corresponding prediction patches;
Updating a reconstruction model based on the training loss. The method according to claim 2, wherein the calculation of the prediction patch, the calculation of the training loss, and the update of the reconstruction model are repeated until the training converges. The method of claim 2, wherein in training the reconstruction model, the one or more patch bounding boxes of the reference image are determined based on keypoints detected in the reference image. Generating an image corresponding to the query image,
Determining one or more patch bounding boxes in the query image;
Calculating a predicted patch for each of the one or more patch bounding boxes of the query image using the reconstruction model;
Calculating a prediction error for the prediction patch based on the one or more patch bounding boxes and corresponding prediction patches of the query image;
Reconstructing the query image by selecting a predictive pixel value at each pixel location based on the predictive patch and the corresponding predictive error. The predicted pixel value is
The method according to claim 5, wherein the prediction patch having the smallest prediction error is predicted. The calculation of the prediction patch is
The method of claim 2, wherein features are extracted from an annular region surrounding each of the one or more patch bounding boxes of the reference image and calculated based on the extracted features. Furthermore,
8. The method of claim 7, comprising pre-processing the reference image before extracting the features so that the features are extracted from the annular region. The calculation of the prediction patch is
The method of claim 5, wherein the method is calculated based on features extracted from an annular region surrounding each of the one or more patch bounding boxes of the query image. Furthermore,
The method of claim 9, comprising pre-processing the query image before extracting the features so that the features are extracted from the annular region. The method of claim 1, wherein the reference image is a plurality of images of the scene captured from different viewpoints. The method of claim 1, wherein the reconstruction model comprises an inverse Gaussian filter. A program for causing a computer to execute the method according to any one of claims 1 to 12. A computer-readable storage medium storing the program according to claim 13.

Images