KR101826463B1 - Method and apparatus for synchronizing time line of videos - Google Patents

Abstract

Provided are a method and an apparatus for synchronizing a time axis of a video. In order to synchronize a time axis of videos, a reference sequence of a reference video is determined and a target sequence of a target video corresponding to the reference sequence is determined. Time axes of the reference video and the target video are synchronized based on the reference sequence and the target sequence.

Description

METHOD AND APPARATUS FOR SYNCHRONIZING TIME LINE OF VIDEOS BACKGROUND OF THE INVENTION [0001]

The present invention relates to a technique for adjusting a time axis of a moving image, and more particularly to an apparatus and method for adjusting a time axis of moving images captured by a plurality of cameras.

Digital contents are being developed in various forms such as 3D image and panoramic image. Among them, panoramic images are contents that provide a wide viewing angle by matching various videos, and there are panoramic still images and panoramic videos. Panoramic still images are created by matching several still images into one still image, which is usually aimed at a static background. Panoramic video is made of one panoramic video by matching videos shot by multiple cameras, which is technically more difficult than panoramic still images.

Korean Patent Publication No. 10-2016-0021501 (published on February 26, 2016) discloses a video processing apparatus. A disclosed moving image processing apparatus includes a storage unit for storing a moving image photographed by a camera, a target object in a plurality of key frames included in the moving image, and a background image, respectively, separates each separated background image, And a controller for acquiring a panoramic image by synthesizing the panoramic image and generating a panoramic image using the obtained panoramic image.

One embodiment may provide an apparatus and method for synchronizing the temporal axis of a moving picture.

One embodiment can provide an apparatus and method for generating a panoramic moving picture using a plurality of moving pictures.

According to an aspect, a moving picture time axis synchronization method includes: determining a reference sequence of one or more sequences in a reference moving picture; Extracting at least one feature point of the reference sequence based on the reference sequence and the target moving picture; Determining a target sequence in the target moving picture corresponding to the reference sequence based on the feature point; And synchronizing the time axis of the reference moving picture and the time axis of the target moving picture based on the reference sequence and the target sequence.

The reference moving image and the target moving image may be photographed using cameras different from each other.

The moving picture time axis synchronization method may further include dividing frames in the reference moving picture into the one or more sequences, wherein each of the one or more sequences may include a predetermined number of frames.

Wherein the step of determining a reference sequence among the one or more sequences in the reference moving image comprises: calculating a sequence similarity for each of the one or more sequences; And determining the reference sequence based on the sequence similarity.

Wherein the step of calculating the sequence similarity comprises: generating a histogram of each of the frames in the first sequence; Calculating a similarity between a histogram of a first frame in the first sequence and a histogram of a second frame temporally adjacent to the first frame; And calculating the sequence similarity based on the calculated similarity.

The step of extracting the one or more feature points may comprise extracting the one or more feature points using a Speed Up Robust Features (SURF) algorithm based on the frames in the reference sequence and the frames in the target motion picture .

Wherein determining the target sequence comprises: calculating a similarity between each of the one or more sequences in the reference sequence and the target moving picture; And determining the target sequence of one or more sequences in the target moving image based on the calculated similarity.

The moving picture time axis synchronization method may further include generating a panoramic moving picture using the reference moving picture and the target moving picture whose time axis is synchronized.

According to another aspect, a moving picture time axis synchronization apparatus includes: a memory in which a program for synchronizing a time axis of a moving picture is recorded; And a processor for performing the program, the program comprising: determining a reference sequence of one or more sequences in a reference moving picture; Extracting at least one feature point of the reference sequence based on the reference sequence and the target moving picture; Determining a target sequence in the target moving picture corresponding to the reference sequence based on the feature point; And synchronizing a time axis of the reference moving picture and the time axis of the target moving picture based on the reference sequence and the target sequence.

The reference moving image and the target moving image may be photographed using cameras different from each other.

The program further comprises dividing frames in the reference moving picture into the one or more sequences, wherein each of the one or more sequences may include a predetermined number of frames.

Wherein the step of determining a reference sequence among the one or more sequences in the reference moving image comprises: calculating a sequence similarity for each of the one or more sequences; And determining the reference sequence based on the sequence similarity.

Wherein the step of calculating the sequence similarity comprises: generating a histogram of each of the frames in the first sequence; Calculating a similarity between a histogram of a first frame in the first sequence and a histogram of a second frame temporally adjacent to the first frame; And calculating the sequence similarity based on the calculated similarity.

The step of extracting the one or more feature points may comprise extracting the one or more feature points using a Speed Up Robust Features (SURF) algorithm based on the frames in the reference sequence and the frames in the target motion picture .

Wherein determining the target sequence comprises: calculating a similarity between each of the one or more sequences in the reference sequence and the target moving picture; And determining the target sequence of one or more sequences in the target moving image based on the calculated similarity.

The program may further comprise generating a panoramic moving picture using the reference moving picture and the target moving picture whose time axis is synchronized.

An apparatus and method for synchronizing the time axis of a moving picture is provided.

An apparatus and method for generating a panoramic moving picture using a plurality of moving pictures are provided.

1 shows a moving picture photographing system according to an example.
2 is a configuration diagram of a moving picture time axis synchronization apparatus according to an embodiment.
3 is a flowchart of a moving picture time axis synchronization method according to an embodiment.
4 is a flow diagram of a method for determining a reference sequence according to an example.
5 is a flowchart of a method of calculating a sequence similarity according to an example.
6 is a flow diagram of a method for determining a target sequence in accordance with an example.
FIG. 7 illustrates a method for determining a target sequence according to an example.

In the following, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the patent application is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

Various modifications may be made to the embodiments described below. It is to be understood that the embodiments described below are not intended to limit the embodiments, but include all modifications, equivalents, and alternatives to them.

The terms used in the examples are used only to illustrate specific embodiments and are not intended to limit the embodiments. The singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, the terms "comprises" or "having" and the like refer to the presence of stated features, integers, steps, operations, elements, components, or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this embodiment belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

In the following description of the present invention with reference to the accompanying drawings, the same components are denoted by the same reference numerals regardless of the reference numerals, and redundant explanations thereof will be omitted. In the following description of the embodiments, a detailed description of related arts will be omitted if it is determined that the gist of the embodiments may be unnecessarily blurred.

1 shows a moving picture photographing system according to an example.

Each of the cameras 120 and 130 creates images by shooting a scene 110. The plurality of images 124 captured by the first camera 120 may constitute a moving image at a first time point. The plurality of images 134 photographed by the second camera 130 may constitute a moving image at the second time point.

When the field of view 122 of the first camera 120 overlaps the field of view 132 of the second camera 130, a panoramic image is created using the images 124 and 134 taken by the respective cameras. A video can be created. To create a panoramic movie, a panoramic image must first be created. A panoramic image can be created by stitching two or more images taken at the same time.

The images 124 and 134 taken using different cameras 120 and 130 may have different time axes. For example, there may be differences in the internal clocks (or clocks) of each of the cameras 120 and 130 such that the images 124 and 134 may have different time axes. When synchronizing the time axis of images 124 and 134, a more enhanced panoramic image can be generated.

Hereinafter, a method of synchronizing the time axis of a moving picture will be described in detail with reference to FIG. 2 to FIG.

2 is a configuration diagram of a moving picture time axis synchronization apparatus according to an embodiment.

The moving picture time axis synchronizer 200 (hereinafter, the moving picture time axis synchronizer 200 is outlined as an apparatus 200) includes a communication unit 210, a processor 220 and a memory 230. According to an embodiment, the apparatus 200 may further include a plurality of cameras (not shown). For example, the device 200 may be implemented as a system-on-chip (SOC), but is not limited to the described embodiments.

The communication unit 210 is connected to the processor 220 and the memory 230 to transmit and receive data. The communication unit 210 may be connected to another external device to transmit / receive data.

The communication unit 210 may be implemented as a circuitry in the device 200. For example, the communication unit 210 may include an internal bus and an internal bus. As another example, the communication unit 210 may be an element that connects the device 200 and an external device. The communication unit 210 may be an interface. The communication unit 210 can receive data from an external device and transmit data to the processor 220 and the memory 230. [

The processor 220 processes the data received by the communication unit 210 and the data stored in the memory 230. "Processor" may be a data processing device embodied in hardware having circuitry having a physical structure for performing desired operations. For example, the desired actions may include code or instructions included in the program. For example, a data processing apparatus embodied in hardware may be a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, , An application-specific integrated circuit (ASIC), and a field programmable gate array (FPGA).

The processor 220 executes computer readable code (e.g., software) stored in a memory (e.g., memory 230) and instructions triggered by the processor 220.

The memory 230 stores data received by the communication unit 210 and data processed by the processor 220. [ For example, the memory 230 may store a program. The stored program may be a set of syntaxes executable by the processor 220 that are coded to synchronize the time axis of the moving image (or image).

According to an aspect, the memory 230 may include one or more volatile memory, non-volatile memory and random access memory (RAM), flash memory, hard disk drive, and optical disk drive.

The memory 230 stores a set of instructions (e.g., software) that operate the device 200. The instruction set that operates the device 200 is executed by the processor 220.

Each of the plurality of cameras generates an image by shooting a scene.

The communication unit 210, the processor 220, and the memory 230 will be described in detail with reference to Figs. 3 to 7 below.

3 is a flowchart of a moving picture time axis synchronization method according to an embodiment.

The following steps 310 to 370 are performed by the device 200 described above with reference to Fig.

In step 310, the communication unit 310 acquires a reference moving picture and a target moving picture. For example, the reference moving image corresponds to the plurality of images 124 described above with reference to FIG. 1, and the target moving image may correspond to the plurality of images 134. The reference video and the target video may be videos taken by different cameras.

According to an aspect, when the apparatus 200 includes a plurality of cameras, the communication unit 310 can acquire the reference moving image and the target moving image by loading images generated by the camera.

According to another aspect, the communication unit 310 can acquire the reference moving image and the target moving image through an external device.

At step 320, the processor 220 divides the frames in the motion picture into one or more sequences. For example, processor 220 may divide frames within a reference moving picture into one or more sequences. The sequence may include a predetermined number of frames. For example, each sequence may include five frames in the order of generation of the frames.

At step 330, the processor 220 determines a reference sequence of one or more sequences in the reference moving picture. The reference sequence may be a sequence used to synchronize the time axis of the reference moving image with the target moving image. The reference sequence may be the sequence with the greatest variation within the frame of one or more sequences. If there are many changes in the frame, frame matching of the reference moving image and the target moving image may be easy.

A method of determining the reference sequence will be described in detail below with reference to FIGS.

At step 340, the processor 220 extracts one or more feature points of the reference sequence. The feature point may be used to determine a sequence of target videos corresponding to the reference sequence.

According to an aspect, the processor 220 may extract one or more feature points using a Speed Up Robust Features (SURF) algorithm based on the frames in the reference sequence and the frames in the target video.

In step 350, the processor 220 determines a target sequence of target videos corresponding to the reference sequence based on the feature points. For example, if the target video is divided into one or more sequences, the target sequence of one or more sequences may be determined. Each of the sequences of the target moving picture may include the same number of frames as the sequence of the reference moving picture. A method of determining a target sequence will be described in detail below with reference to Fig.

At step 360, the processor 220 synchronizes the time axis of the reference video and the target video based on the reference sequence and the target sequence. For example, the processor 220 can synchronize the time axis of the reference moving image and the target moving image by setting frames of the reference sequence and frames of the target sequence to be photographed at the same time.

Step 370 may optionally be performed after step 360 has been performed.

In step 370, the processor 220 generates a panoramic movie using the reference moving picture and the target moving picture whose time axis is synchronized. The processor 220 may generate a panoramic image for a specific time using a frame in the reference moving image and a frame of the target moving image and generate a panoramic image based on the panoramic images.

4 is a flow diagram of a method for determining a reference sequence according to an example.

Step 330 described above with reference to FIG. 3 includes the following steps 410, 420.

At step 410, the processor 220 calculates a sequence similarity for each of the one or more sequences of the reference moving picture. The sequence similarity may indicate the similarity between the frames included in the sequence. The lower the sequence similarity is, the more the object of the corresponding sequence is determined to have a lot of motion. A method of calculating the sequence similarity will be described in detail below with reference to FIG.

At step 420, the processor 220 determines a reference sequence based on the sequence similarity. For example, a sequence with the lowest sequence similarity may be determined as a reference sequence.

5 is a flowchart of a method of calculating a sequence similarity according to an example.

Step 410, described above with reference to FIG. 4, includes the following steps 510, 520, 430.

At step 510, the processor 220 generates a histogram for each of the frames in the first sequence of reference animations. The first sequence means one of more than one sequences of the reference moving picture, and does not represent a specific sequence. The histogram may be a distribution of R (Red), G (Green), and B (Blue) values of each pixel included in the frame.

At step 520, the processor 220 calculates the similarity between the histogram of the first frame in the first sequence and the histogram of the second frame temporally adjacent to the first frame. The first frame and the second frame are both frames in the first sequence. For example, if the frame number of the first frame is 45, the frame number of the second frame may be 46.

At step 530, the processor 220 calculates the sequence similarity of the first sequence based on the calculated similarity. For example, if the first sequence includes n frames, the sequence similarity of the first sequence may be calculated using Equation (1) below. The similarity (i, i + 1) is the similarity between the i-th frame and the (i + 1) -th frame.

6 is a flow diagram of a method for determining a target sequence in accordance with an example.

Step 350 described above with reference to FIG. 3 includes the following steps 610 and 620.

In step 610, the processor 220 calculates the similarity between each of the sequences in the reference sequence and the target moving picture. The processor 220 can calculate the similarity between the reference sequence and the sequence of the target moving image using the minutiae points. The processor 220 may perform image matching for each of the sequences of the reference sequence and the target moving image. For example, the image matching may be a SURF-based Descriptor Matching. Processor 220 may calculate nearest point information of a target frame for each feature point using descriptor matching. The target frame is a frame of the target video in which image matching is performed. The processor 220 may calculate the Euclidean distance for the feature point and the nearest point based on the nearest point of contact information. The Euclidean distance can correspond to the degree of similarity.

In step 620, the processor 220 determines a target sequence of one or more sequences in the target moving image based on the calculated similarity between the sequences. For example, the processor 220 may determine the sequence with the highest degree of similarity as the target sequence.

The processor 220 may synchronize the time axis of the reference video and the target video based on the reference sequence and the target sequence, if a target sequence is determined. For example, the reference sequence and the target sequence may be set to be photographed at the same time, and the sequences of the target moving image may be matched with the sequences of the reference moving image according to the order of the sequence.

FIG. 7 illustrates a method for determining a target sequence according to an example.

7, the reference moving picture 710 and the target moving picture 720 are intuitively shown. The reference moving image 710 includes 300 frames, and the frames may be divided into one or more sequences. A reference sequence 712 of the sequences may be determined based on the sequence similarity.

The target moving image 720 may include a plurality of frames. For example, the target moving image 720 may include the same number of frames as the reference moving image 710. [ As another example, the target moving image 720 may include a different number of frames than the reference moving image 710. [

The frames of the target moving image 720 may be divided into one or more sequences, and the number of frames included in each sequence may be the same as the number of frames included in the sequence of the reference moving image 710.

The similarity of each of the sequences of the reference sequence 712 and the target moving image 720 is calculated, and the target sequence is determined based on the similarity. If the target sequence is determined, the time axis of the target sequence can be synchronized to be located on the same time axis as the reference sequence.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described with reference to the drawings, various technical modifications and variations may be applied to those skilled in the art. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Embodiments may include a computer readable medium having program instructions for performing various computer implemented operations. In particular, in the present embodiment, a content display code for displaying digital content on a screen of the touch-sensitive display, an object recognition code for recognizing an object selected by the user in the digital content through a touch input of the user for the touch- An annotation storing code for generating and storing annotation data in which an annotation is applied to an object in response to a request, an annunciation count code for counting the number of annotation data generated in each item for each item in which the contents of the digital contents are classified, , And an engineer providing code for providing the number of annotation data items per item at the request of the user. The program according to the present embodiment may be composed of a PC-based program or a mobile terminal exclusive application (for example, a form of a smart phone application, a feature phone VM (virtual machine), etc.). This may be configured to be executed by the user terminal and / or at least one processor stored in the memory of the electronic book service system and / or running in the user terminal and / or the electronic book service system.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

200: Video time axis synchronization device
210:
220: Processor
230: Memory

Claims (17)

Determining a reference sequence of one or more sequences in the reference moving picture;
Extracting at least one feature point of the reference sequence based on the reference sequence and the target moving picture;
Determining a target sequence in the target moving picture corresponding to the reference sequence based on the feature point; And
Synchronizing the time axis of the reference moving picture and the time axis of the target moving picture based on the reference sequence and the target sequence
Lt; / RTI >
Wherein determining a reference sequence of one or more sequences in the reference moving picture comprises:
Generating a histogram of each of the frames in the first sequence;
Calculating a similarity between a histogram of a first frame in the first sequence and a histogram of a second frame temporally adjacent to the first frame;
Calculating a sequence similarity for the first sequence based on the calculated similarity; And
Determining a sequence having the lowest sequence similarity among the one or more sequence similarities calculated for the one or more sequences as the reference sequence;
/ RTI >
How to synchronize video time axis.
The method according to claim 1,
Wherein the reference moving image and the target moving image are photographed using cameras different from each other,
How to synchronize video time axis.
The method according to claim 1,
Dividing frames in the reference moving picture into the one or more sequences
Further comprising:
Wherein each of the one or more sequences comprises a predetermined number of frames,
How to synchronize video time axis.
delete delete The method according to claim 1,
Wherein the extracting of the one or more feature points comprises:
Extracting the one or more feature points using a SURF (Speeded Up Robust Features) algorithm based on frames in the reference sequence and frames in the target video
/ RTI >
How to synchronize video time axis.
The method according to claim 1,
Wherein determining the target sequence comprises:
Calculating a similarity between each of the one or more sequences in the reference sequence and the target moving picture; And
Determining the target sequence of one or more sequences in the target moving picture based on the calculated similarity
/ RTI >
How to synchronize video time axis.
The method according to claim 1,
Generating a panoramic moving picture using the reference moving picture and the target moving picture whose time axis is synchronized
≪ / RTI >
How to synchronize video time axis.
A computer-readable recording medium embodying a program for carrying out the method according to any one of claims 1 to 3 and 6 to 8.
An apparatus for synchronizing a time axis of a moving picture,
A memory in which a program for synchronizing a time axis of a moving picture is recorded; And
The processor
Lt; / RTI >
The program includes:
Determining a reference sequence of one or more sequences in the reference moving picture;
Extracting at least one feature point of the reference sequence based on the reference sequence and the target moving picture;
Determining a target sequence in the target moving picture corresponding to the reference sequence based on the feature point; And
Synchronizing the time axis of the reference moving picture and the time axis of the target moving picture based on the reference sequence and the target sequence
Lt; / RTI >
Wherein determining a reference sequence of one or more sequences in the reference moving picture comprises:
Generating a histogram of each of the frames in the first sequence;
Calculating a similarity between a histogram of a first frame in the first sequence and a histogram of a second frame temporally adjacent to the first frame;
Calculating a sequence similarity for the first sequence based on the calculated similarity; And
Determining a sequence having the lowest sequence similarity among the one or more sequence similarities calculated for the one or more sequences as the reference sequence;
/ RTI >
Video time axis synchronization device.
11. The method of claim 10,
Wherein the reference moving image and the target moving image are photographed using cameras different from each other,
Video time axis synchronization device.
11. The method of claim 10,
The program includes:
Dividing frames in the reference moving picture into the one or more sequences
Lt; / RTI >
Wherein each of the one or more sequences comprises a predetermined number of frames,
Video time axis synchronization device.
delete delete 11. The method of claim 10,
Wherein the extracting of the one or more feature points comprises:
Extracting the one or more feature points using a SURF (Speeded Up Robust Features) algorithm based on frames in the reference sequence and frames in the target video
/ RTI >
Video time axis synchronization device.
11. The method of claim 10,
Wherein determining the target sequence comprises:
Calculating a similarity between each of the one or more sequences in the reference sequence and the target moving picture; And
Determining the target sequence of one or more sequences in the target moving picture based on the calculated similarity
/ RTI >
Video time axis synchronization device.
11. The method of claim 10,
The program includes:
Generating a panoramic moving picture using the reference moving picture and the target moving picture whose time axis is synchronized
Lt; / RTI >
Video time axis synchronization device.

Images