KR20130104215A

KR20130104215A - Method for adaptive and partial replacement of moving picture, and method of generating program moving picture including embedded advertisement image employing the same

Info

Publication number: KR20130104215A
Application number: KR1020120025536A
Authority: KR
Inventors: 김종현
Original assignee: 계원예술대학교 산학협력단; 김종현
Priority date: 2012-03-13
Filing date: 2012-03-13
Publication date: 2013-09-25

Abstract

PURPOSE: An adaptive partial image replacement method and a video image producing method with an inserted advertisement image using the same are provided to allow a viewer to recognize an advertisement image as a part of an original image by replacing a part of the original image with a replacement image and adaptively modifying the location and size of the replacement image based on the change of the original image. CONSTITUTION: An original image input unit obtains an original image from a camera or a first storage unit (200). A user sets up a replacement area and a replacement section through a user interface unit (202). If the user presses a conversion start button, a feature extraction unit extracts a perspective feature from the original image. A warping unit warps and adjusts a replacement image in accordance with the perspective feature. The warping unit adjusts the boundary of the replacement area. An image synthesis unit combines the replacement image processed by the warping unit with the original image (206). [Reference numerals] (200) Obtain an original image; (202) Set up a replacement area and a replacement section; (204) Move to the first frame of the replacement section; (206) Replace an image of the replacement section in the first frame with an alternative image; (208) Move to the next from; (210) Replace an image of the replacement section in the current frame with an alternative image; (212) Is the current frame the last frame of the replacement section ?; (AA) Start; (BB) No; (CC) Yes; (DD) End

Description

Method for Adaptive and Partial Replacement of Moving Picture, and Method of Generating Program Moving Picture Including Embedded Advertisement Image Employing the Same}

The present invention relates to an image processing method, and more particularly, to a method of replacing a part of a video with another image.

There are many areas where you need to replace some of your videos with other videos. For example, Korean Patent Laid-Open Publication No. 10-2009-0087774 discloses a text, picture, or motion data by recognizing a character, a picture, a caller's motion in a video call frame during a video call using a portable terminal, and extracting the extracted data in advance. A system for recognizing and converting image data, which is converted into still image data or moving image data, synthesized or replaced with a video call frame, and delivered to a called terminal. In addition, Korean Patent Application Publication No. 10-1050107 describes an image control apparatus that replaces an image of a specific person in a karaoke image or a UCC video with a character and controls the movement and movement of the character so that the character moves according to the movement of the character.

In particular, the inventors are interested in the field of program advertising. That is, in a program image that is previously transmitted to a user through airwaves, cable networks, the Internet, or other media, an advertisement is provided to the user at the start and end of the program or through an intermediate advertisement. In the case of advertisements provided at the start and end of the program, the viewers often do not pay attention or stop watching the video. Therefore, it is difficult to attract the viewer's attention and thus the advertising effect is insignificant. In the case of interstitial advertisements, the concentration of the program decreases, which not only causes viewer dissatisfaction, but also the effect of advertising is not high due to the high possibility of changing channels during the advertisement time. In addition, in some cases, an advertisement is inserted in the form of a still image using a subtitle or a figure at the bottom of the program.

Therefore, in order to enhance the advertising effect, a method of naturally inserting an advertisement image into the program is required so as to attract the viewer's attention without interrupting the flow of viewing the program.

SUMMARY OF THE INVENTION The present invention has been made in an effort to solve such a problem, and provides a method of replacing an image by replacing an image with another image and adaptively changing the position and size of the replacement image according to the change of the original image. Shall be.

In addition, the present invention, by applying the image replacement method as described above in the advertising field, a part of the program original image is replaced with the advertisement image, but the image that can adaptively change the position and size of the advertisement image in accordance with the change of the original image It is another technical problem to provide a generating method.

In addition, another technical problem of the present invention is to provide a program suitable for implementing a program image generating method in which an advertisement image is inserted as described above.

In the image replacement method of the present invention for achieving the above technical problem, first, a replacement image is prepared. In addition, a user input for a replacement area and a replacement section to be replaced with the replacement image in the original image is accepted. Subsequently, perspective characteristics are extracted from the original image, the replacement image is warped and adjusted according to the perspective characteristics, and the boundary of the replacement region is adjusted. Finally, the warped replacement image is synthesized with the original image.

For the second and subsequent frames, in adjusting the replacement image according to the perspective characteristics, it is desirable to readjust the replacement image and the replacement area when the object moving distance is large while continuously checking the object moving distance and direction from the previous frame. . To this end, first, a moving distance of a predetermined region is extracted between the previous frame and the current frame of the original image. When the moving distance is out of a predetermined threshold value, a homography matrix between the previous frame and the current frame is determined, and the position of the replacement area is changed. Subsequently, the perspective image is re-extracted in the original image, and the replacement image is warped and adjusted to match the re-extracted perspective characteristic.

Meanwhile, in the program image generating method in which the advertisement image of the present invention is inserted to achieve the above technical problem, first, the advertisement image is prepared. In addition, the user input for the replacement area and the replacement section to be replaced with the advertisement image in the original program image is accepted. Subsequently, perspective characteristics are extracted from the original program image, warping and adjusting the advertisement image according to the perspective characteristics, and the boundary of the substitute area is adjusted. Finally, the warped advertisement image is synthesized with the program original image.

For the second and subsequent frames, in adjusting the advertisement image according to the perspective characteristics, it is desirable to readjust the advertisement image and the replacement area when the object movement distance is large while continuously checking the object movement distance and direction from the previous frame. . To this end, first, a moving distance of a predetermined region is extracted between the previous frame and the current frame of the original program image. When the moving distance is out of a predetermined threshold value, a homography matrix between the previous frame and the current frame is determined, and the position of the replacement area is changed. Subsequently, the perspective characteristics are re-extracted in the original program image, and the advertisement image is warped and adjusted to match the re-extracted perspective characteristics.

On the other hand, the program of the present invention can be stored and executed in a computer-readable recording medium, comprising: (a) a replacement area for receiving a program original image and an advertisement image and replacing the advertisement image within the program original image; (B) extracting perspective characteristics from the original image of the program, warping and adjusting the advertisement image according to the perspective characteristics, and adjusting the boundary of the replacement region; , (c) is suitable for implementing a function of synthesizing the warped advertisement image to the program original image.

In particular, the (b) function includes (b1) a function of extracting a moving distance of a predetermined region between a previous frame and a current frame of the original program image, and (b2) when the moving distance is out of a predetermined threshold value, Determining a homography matrix between a frame and the current frame, changing a position of the replacement area, (b3) re-extracting perspective characteristics within the original program image, and reconstructing the advertisement image according to the re-extracted perspective characteristics It includes the ability to warp and adjust.

As described above, the present invention replaces a part of the program original image with the replacement image, and adaptively changes the position and size of the replacement image according to the change of the original image.

Accordingly, when the method of the present invention is applied to the program advertisement field, when the original program image is displayed on the viewer's display device, the advertisement image can be felt as if it exists in the original image, and the program viewing image is not disturbed. The effect is that the original program video can be viewed.

On the other hand, since the advertisement video can be naturally displayed to the viewer while the original program video is playing, it is possible to prevent the viewer from avoiding the advertisement video as much as possible, and maximize the advertising effect from the program producer or advertiser's point of view. There is an advantage that it can.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the figure,
1 is a block diagram of an image replacement apparatus according to the present invention.
FIG. 2 is a detailed block diagram of the image replacement program shown in FIG. 1.
FIG. 3 is a diagram illustrating an example of an area / section setting screen provided by the user interface shown in FIG. 2.
4 is a flowchart illustrating an embodiment of an overall execution process of an image replacement method according to the present invention.
FIG. 5 is a flowchart illustrating in detail an image replacement step for an initial frame, that is, step 206 shown in FIG. 4.
FIG. 6 is a diagram for describing an example of a process of adjusting a substitute area in FIG. 5.
FIG. 7 is a flowchart illustrating in detail a replacement step for a frame after the first frame, that is, step 210 shown in FIG. 4.

Referring to FIG. 1, an image replacement apparatus according to an exemplary embodiment of the present invention may include a camera 100, a first storage unit 102, an image data processor 110, a mouse 120, a keyboard 122, and a transmitter. 130, and a second storage unit 132.

The camera 100 captures a program image and outputs a real-time program image. The first storage unit 102 may store a program image already photographed by the camera 100 and output the stored program image in non-real time. In addition, the first storage unit 102 stores a substitute image, for example, an advertisement image, previously selected by the device user.

The image data processing unit 110 is a real-time program image output from the camera 100 or a non-real-time program read out from the first storage unit 102 as set and operated by the mouse 120 and / or the keyboard 122. In the image (hereinafter referred to as the original image), a part is replaced with an alternate image. In order to perform the replacement process, the image data processing unit 110 executes the image replacement program according to the present invention. The image data processing unit 110 may be implemented by a conventional microprocessor or a combination of a separate video processor and a microprocessor.

The program image, which is partially replaced by the substitute image by the image data processing unit 110, is provided to the viewer through the transmitting unit 130, or stored in the second storage unit 132 and later provided to the viewer.

In FIG. 1, only one of the camera 100 and the first storage unit 102 may be selectively used, and any one of the transmitter 130 and the second storage unit 132 may be omitted. In addition, the first storage unit 102 and the second storage unit 132 may be implemented in the same physical device.

Referring to FIG. 2, the image replacement program includes an original image input unit 150, a decoder 152, a user interface unit 154, an area / section data buffer 156, a feature extractor 158, and an alternative image reading unit ( 160, a warping unit 162, an image synthesizing unit 164, and an encoder 166.

The original image input unit 150 receives the original image. That is, the original image input unit 150 receives the real-time program image from the camera 100 or executes the non-real time stored in the first storage unit 102 when the image replacement function execution is selected through the user interface unit 154. Read the program video. The decoder 152 is selectively used, and decompresses and descrambles an RGB image frame when the original image is formatted in a specific video format such as MPEG-2 or MPEG-4.

The user interface unit 154 may instruct the user to execute and end the image replacement function, and set an image area to be replaced with the replacement image and a time period to be replaced (hereinafter, referred to as an “substitution interval”). 3 shows an example of an area / section setting screen. As shown, the setting screen is provided with an image display area 180 for displaying the original image frame, a play / pause button 182, a stop button 184, a Fast Forward button 186, and a Rewind button 188. The frame rate input window 190 is provided so that the playback frame rate can be arbitrarily set. In addition, the setting screen includes frame selection buttons 192 and 194 for selecting the start and end frames of the replacement section, vertex selection function buttons 196 and 198 for selecting diagonal vertices of the replacement area, and a conversion start button. 199 is provided.

The area / section data buffer 156 stores data about the replacement area and the replacement section set through the user interface unit 154. In addition, the area / section data buffer 156 also stores replacement area data adjusted by the feature extraction unit 158 as described later.

The feature extractor 158 extracts a feature around the substitute area, for example, a perspective feature, in the image. In addition, the feature extractor 158 extracts a moving distance and a direction of the replacement region between successive frames and calculates a homography matrix. In addition, the feature extractor 158 may adjust the replacement area based on the homography matrix, and store the adjusted replacement area data in the area / section data buffer 156.

The substitute image reading unit 160 reads the substitute image selected by the user through the user interface unit 154 from the plurality of substitute images stored in the first storage unit 102.

The warping unit 162 warps the substitute image based on the feature around the substitute area extracted by the feature extractor 158. The image synthesizing unit 164 synthesizes the warped replacement image in the original image, such that the replacement area portion of the original image is replaced by the warped replacement image. The encoder 166 may be provided corresponding to the decoder 152 and formats the image output from the image synthesizing unit 164 in the same format as the original image.

Hereinafter, the operation of the image replacement apparatus shown in FIG. 1 will be described with reference to FIGS. 4 to 7.

Figure 4 shows the overall implementation of the image replacement method according to the present invention.

First, the original image input unit 150 obtains an original image from the camera 100 or the first storage unit 102 (operation 200).

In operation 102, the user sets the replacement region and the replacement section through the user interface 154 (operation 202). That is, the user inputs the frame rate in the frame rate input window 190 in the setting screen shown in FIG. 3, while operating the buttons 182 to 184 while watching the original image displayed in the image display area 180. It is possible to search for parts to be set as replacement areas and replacement sections. For example, the user may input a frame rate lower than 30 frames / s, which is a normal frame rate, for example, 3 frames / s into the frame rate input window 190 to search for an alternative section while changing the frame gradually. have.

In the preferred embodiment, the selection of the alternative section is made using the start button 192 and the end button 194. When the user presses the start button 192 while the original image is displayed in the image display area 180, the user interface 154 uses the frame time information or the frame number at this time as the start identification data of the replacement section. The data is stored in the data buffer 156. In addition, when the user presses the end button 194, the user interface unit 154 stores the frame time information or the frame number at this time in the area / section data buffer 156 as end identification data of the replacement section.

In one embodiment, after the user presses the end button 194, the user interface unit 154 displays the first frame of the replacement section in the image display area 180. In this state, the user can press the vertex selection function buttons 196 and 198 to select the diagonal vertex of the replacement area. That is, when a user clicks a button in the image display area 180 with the mouse 120 after pressing the button 196, the corresponding point is selected as one vertex of the replacement area, and after pressing the button 198 again, When another point in the image display area 180 is clicked with the mouse 120, the corresponding point is selected as another vertex of the substitute area. Accordingly, a rectangular region having two vertices as diagonal vertices is determined as the replacement region. The determined replacement area data is stored in the area / section data buffer 156. In this case, the user may press the vertex selection function buttons 196 and 198 again to adjust the replacement area.

Thereafter, when the user presses the conversion start button 199, an image replacement process is started by the feature extractor 158, the warping unit 162, and the image synthesizing unit 164. First, after the frame movement is made to the first frame in the replacement section (step 204), the image in the replacement area is replaced with the replacement image in the first frame (step 206). Subsequently, frame movement is performed to the next frame (step 208), and the image in the replacement area in the current frame is replaced with the replacement image (step 210).

Subsequently, after determining whether the current frame is the last frame in the replacement section (step 212), if it is not the last frame, the flow returns to step 208 to continue image replacement for the next frame. On the other hand, if it is determined in step 210 that the current frame is outside the range of the replacement section, the process ends.

5 shows in more detail the alternative step, 206, for the original frame.

First, the feature extractor 158 extracts perspective characteristics in the original image (step 220), and re-determines four corners of the replacement area according to the extracted perspective characteristics (step 222). For example, assuming that the image displayed on the image display area 180 is the same as that of FIG. 6 and the replacement area is set to REGION1, the feature extractor 158 extracts two or more horizontal lines. In the drawing, the ceiling line LINE1 and the back line LINE2 of the table may be extracted as reference lines representing perspective characteristics. In this state, the feature extractor 158 redefines the upper line LINE3 and the lower line LINE4 of the replacement region REGION1 from the two reference lines by the law of the proportional part. In addition, the point P1 to P4 where the upper line LINE3 and the lower line LINE4 meet with the vertical lines on both sides of the replacement area REGION1 are determined as vertices for redefining the replacement area, and the area / section is adjusted as the adjusted replacement area data. The data is stored in the data buffer 156.

In operation 224, the warping unit 224 warps the replacement image read from the first storage unit 102 by the substitute image reading unit 160 according to the perspective characteristics. In one embodiment, the warping unit 224 adjusts the image so that the upper and lower edges of the substitute image correspond to the upper line LINE3 and the lower line LINE4, respectively, and adjust the horizontal size accordingly.

In operation 226, the image synthesizing unit 164 synthesizes the warped replacement image to the original image, thereby replacing the replacement region part of the original image by the warped replacement image.

7 shows in more detail an alternative step, ie step 210, for a frame after the first frame.

After the first frame, the camera angle may be changed to change the direction or size of the original image. Accordingly, in the preferred embodiment, the replacement area is changed for the frame after the first frame by comparison with the previous frame.

First, the feature extractor 158 extracts a moving distance and a direction of an object or region around the substitute region by comparing the feature points in the substitute region with the previous frame (step 240). Next, the feature extractor 158 determines whether the movement distance and the direction are within a predetermined threshold value (operation 242). If it is determined that the moving distance and the direction are within the threshold range, the image is synthesized while maintaining the existing substitute area and the warping scale (step 252).

On the other hand, if it is determined in step 242 that the moving distance is out of the threshold range, the feature extractor 158 determines the homography matrix between the previous frame and the current frame and changes the position of the replacement area from the position set by the user. (Step 144).

Next, the feature extractor 158 re-extracts perspective properties in the original image based on the changed replacement area position (step 246), and re-determines four corners of the replacement area according to the extracted perspective properties (step 248). ). The warping unit 224 warps the substitute image read out from the first storage unit 102 by the substitute image reading unit 160 according to the perspective characteristic (operation 250). Finally, in operation 252, the image synthesizing unit 164 synthesizes the replacement image warped to the original image, thereby replacing the replacement region part of the original image by the warped replacement image.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the invention.

For example, in the above embodiment, the coordinates of the points P1 to P4 where the two reference lines LINE3 and LINE4 meet both vertical lines of the replacement area REGION1 are stored in the area / section data buffer 156 as adjusted area data. However, in a modified embodiment, the equations of the two reference lines LINE3 and LINE4 or the coordinates of both endpoints may be stored in the area / section data buffer 156 as adjusted replacement area data. In this case, the warping unit 162 warps the replacement image by referring to the two reference lines LINE3 and LINE4, and the image synthesizing unit 164 synthesizes the images.

On the other hand, in synthesizing the original image and the substitute image, the foreground and background are separated from the original image or the depth of each object is extracted, and if there is an object that covers the area replaced by the substitute image, the part covered by the object is replaced. You can prevent the image from being displayed.

Therefore, it should be understood that the above-described embodiments are illustrative in all aspects and not restrictive. The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

Claims

As a method of replacing part of the original video with the replacement video,
(a) preparing the substitute image;
(b) accepting a user input for a replacement area and a replacement section to be replaced with the replacement image in the original image;
(c) extracting perspective characteristics in the original image, warping and adjusting the replacement image according to the perspective characteristics, and adjusting a boundary of the replacement region; And
(d) synthesizing a warped replacement image with the original image;
It includes, step (c)
extracting a moving distance of a predetermined region between a previous frame and a current frame of the original image;
(c2) determining a homography matrix between the previous frame and the current frame when the moving distance is out of a predetermined threshold value and changing a position of the replacement area; And
(c3) re-extracting perspective characteristics in the original image, and warping and adjusting the replacement image according to the re-extracted perspective characteristics;
Image replacement method comprising a.

As a method of replacing part of the original program video with the advertisement video,
(a) preparing the advertisement image;
(b) accepting a user input for a replacement area and a replacement section to be replaced with the advertisement image in the original program image;
(c) extracting perspective characteristics in the original program image, warping and adjusting the advertisement image according to the perspective characteristics, and adjusting a boundary of the replacement area; And
(d) synthesizing a warped advertisement image to the program original image;
It includes, step (c)
(c1) extracting a moving distance of a predetermined region between a previous frame and a current frame of the original program image;
(c2) determining a homography matrix between the previous frame and the current frame when the moving distance is out of a predetermined threshold value and changing a position of the replacement area; And
(c3) re-extracting perspective characteristics in the original program image and warping and adjusting the advertisement image according to the re-extracted perspective characteristics;
Program image generation method is inserted, the advertising image including a.

(a) receiving a program original image and an advertisement image, and accepting a user input for a replacement area and a replacement section to be replaced with the advertisement image in the program original image;
(b) extracting perspective characteristics from the original program image, warping and adjusting the advertisement image according to the perspective characteristics, and adjusting a boundary of the replacement area; And
(c) a function of synthesizing a warped advertisement image to the program original image;
Can be implemented, and the (b) function
(b1) extracting a moving distance of a predetermined region between a previous frame and a current frame of the original program image;
(b2) determining a homography matrix between the previous frame and the current frame and changing the position of the replacement area when the moving distance is out of a predetermined threshold value; And
(b3) re-extracting perspective characteristics in the original program image and warping and adjusting the advertisement image according to the re-extracted perspective characteristics;
Computer-readable recording medium recording a program comprising a.