KR101870761B1 - System and method for supporting drone imaging by using aviation integrated simulation - Google Patents

Abstract

The storyboard generating system includes a ground control system for transmitting a flight scenario including virtual shooting environment information and shooting mode information related to a UAV to an air graphic simulator, An aerial graphic simulator for controlling the UAV, based on a scenario, for providing the photographed image photographed through the UAV to the ground control system, and a control unit for acquiring the photographed image from the ground control system, And an image storyboard maker for generating a storyboard corresponding to the flight scenario using the selected storyboard image based on the situation condition.

Description

TECHNICAL FIELD [0001] The present invention relates to a system and a method for supporting a drone image capturing using an aviation integrated simulation,

The present invention relates to a system and method for generating a video storyboard based on UAV.

The storyboard plays a role of designing the story, component, shooting information, and copper line to explain the flow of the image. Generally, it is produced by professional art worker using sketch or sketch software. Expert personnel conduct field research before storyboard production, and create storyboards considering the mood and movie story of the location. The film director uses these storyboards to construct the composition of the camera, the distance, the lighting, the motion of the actor, etc., and then starts shooting after confirming that it conforms to the visual aesthetic criteria sought in the film.

On the other hand, in the film-related field, films based on aerial photographing such as airplane action are constantly being produced. Unlike the aerial photographing using an airplane, unlike the ground photographing using a crane or a rail, And the camera mounted on the gimbal can be manipulated in a three-dimensional space, which is advantageous in that it is possible to shoot an image that the film director can not imagine.

However, due to the characteristics of aerial photographing, there are limitations in applying the conventional storyboard making method in view of the object in the air, and there are various variables such as an aircraft controller, a controller, The technique of shooting has a disadvantage in that it can not acquire the composition of the aerial image desired by the film director or the cinematographer. In addition, when the photographing is to be performed again due to NG at the time of aerial photographing, not only a film director and a photographer but also a lot of manpower such as a main actor and an assistant actor are wasted, and time and resources are also wasted.

The background technology of the present application is disclosed in Korean Patent Publication No. 10-2010-0054078 (published on May 24, 2010).

The present invention has been made to solve the above-mentioned problems of the conventional art, and it is an object of the present invention to provide a storyboard creation system and method capable of acquiring a 3D image at an arbitrary angle desired by a film director or a cinematographer at a low cost in a short time, The purpose of this paper is to provide

The present invention has been made to solve the above problems of the conventional art, and it is an object of the present invention to provide a storyboard creation system and method capable of eliminating waste of manpower, time, resources, and the like, The purpose is to provide.

The present invention has been made to solve the above-described problems of the prior art, and it is an object of the present invention to provide an image processing apparatus and a method for acquiring a real- The present invention aims to provide a system and method for creating a storyboard that enables a movie director who does not have knowledge on aviation or a 3D aerial image desired by a cinematographer to more easily acquire a 3D aerial image.

It is an object of the present invention to provide a storyboard creation system and method that can be used as a way point of an actual aircraft (or actual UAV) during actual aerial photographing.

It is to be understood, however, that the technical scope of the embodiments of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

As a technical means for accomplishing the above technical object, a storyboard generating system according to an embodiment of the present invention includes a ground control unit for transmitting a flight scenario including virtual shooting environment information and photographing method information related to a UAV to an air graphic simulator System, an aviation graphic simulator for controlling the UAV based on the flight scenario received from the terrestrial control system, and providing the captured image photographed through the UAV to the terrestrial control system, And an image storyboard maker for generating a storyboard corresponding to the flight scenario using the selected storyboard image based on predetermined conditional conditions from the obtained shot image.

The video storyboard maker may further include a situation setting unit for setting a situation condition corresponding to a characteristic for selecting a scene stored in the storyboard, and a video analysis unit for selecting a storyboard image from the shot video based on the situation condition The situation condition may include a situation variable depending on a degree of importance of a characteristic appearing in each scene included in the photographed image in the selection of the storyboard image.

Also, the UAV may be a virtual UAV flying in the air graphic simulator.

The video storyboard maker may further include a storyboard generating unit for generating the storyboard by combining the storyboard video and the flight information of the UAV at a shooting time corresponding to the storyboard video in the shot video have.

In addition, the flight information of the UAV may include UAV time information corresponding to the shooting point corresponding to the storyboard image in the captured image, and the storyboard generating unit may calculate the UAV time information from the system time Information on the basis of the system time information when it is determined that there is an error in the UAV time information, the error of the UAV time information can be corrected.

In addition, the flight information of the UAV at the time of photographing corresponding to the storyboard image can be used as a way point of the actual UAV when an image is captured through an actual UAV.

In addition, the context condition may include an object context variable for an object included in the captured image.

In addition, the object situation variable may be a variable related to at least one of the property of the object, the number of the object, and the position of the object.

In addition, the conditional condition may include a screen condition variable for a screen change of the photographed image.

In addition, the screen status variable may be a variable related to at least one of whether or not the background is switched and the switching speed of the background.

In addition, the video analysis unit may select the storyboard image considering the plurality of context variables together when a plurality of context variables are included in the context condition.

The video analysis unit may select the storyboard image based on summation information obtained by summing the scalar values of a plurality of context variables for each scene included in the shot image have.

In addition, in the situation setting unit, a maximum number of the storyboard images included in the storyboard is set, and the video analyzer aligns the sum information values calculated for each scene included in the shot image in order of magnitude, And the scenes corresponding to the maximum number can be selected as the storyboard images based on the sorted summation information values.

Meanwhile, the video storyboard maker according to one embodiment of the present invention includes a situation setting unit for setting a situation condition corresponding to a characteristic for selecting a scene stored in a storyboard, The situation condition may include a situation parameter that is assigned according to the importance of a characteristic appearing in each scene included in the shot image when the storyboard image is selected.

In addition, the photographed image may be an image photographed through a UAV.

In addition, the UAV may be a virtual UAV flying in an air graphic simulator.

Meanwhile, the storyboard generating method according to an embodiment of the present invention includes the steps of transmitting, in a ground control system, a flight scenario including virtual shooting environment information and photographing method information related to a UAV to an air graphic simulator, Controlling the UAV based on the flight scenario received from the terrestrial control system and providing the captured image photographed through the UAV to the terrestrial control system, And generating a storyboard corresponding to the flight scenario using the selected storyboard image based on a predetermined situation condition from the obtained shot image.

According to another aspect of the present invention, there is provided a method for producing an image storyboard through a video storyboard maker, the method comprising: setting a situation condition corresponding to a characteristic for selecting a scene to be stored in a storyboard; And selecting a storyboard image from the image, wherein the situation condition may include a situation variable that is assigned according to the importance of a characteristic appearing in each scene included in the shot image when the storyboard image is selected .

The above-described task solution is merely exemplary and should not be construed as limiting the present disclosure. In addition to the exemplary embodiments described above, there may be additional embodiments in the drawings and the detailed description of the invention.

According to the above-mentioned problem solving means of the present invention, based on a situation condition including a situation variable given according to the importance of characteristics appearing in each scene included in a shot image from a shot image shot through a UAV, such as a virtual UAV, A 3D aerial image desired by a film director or a cinematographer who does not have knowledge on aviation can be acquired more easily by selecting a board image and generating a storyboard using the selected storyboard image.

According to the above-mentioned problem solving means of the present invention, a storyboard is created by linking a shot image acquired through a virtual unmanned aerial vehicle with flight information of an unmanned aerial vehicle at the time of shooting the shot image, It can be used as a way point of an aircraft (or an actual UAV) so that a 3D image at an arbitrary angle desired by a film director or a cinematographer can be acquired at a low cost in a short time, And waste of resources and the like can be solved.

1 is a diagram illustrating a schematic configuration of a storyboard generating system according to an embodiment of the present invention.
2 is a flowchart illustrating an operation of a ground control system in a storyboard generating system according to an embodiment of the present invention.
3 is a flowchart illustrating an operation flow of an aviation graphic simulator in a storyboard generating system according to an embodiment of the present invention.
4 is a block diagram schematically illustrating the configuration of an image storyboard maker according to an embodiment of the present invention.
5 is a flowchart illustrating an operation flow of a situation setting unit in an image storyboard maker according to an embodiment of the present invention.
FIG. 6 is a flowchart illustrating an operation of a video analysis unit in an image storyboard maker according to an exemplary embodiment of the present invention.
7 is a flowchart illustrating an operation flow of a storyboard generating unit in the video storyboard producing unit according to an embodiment of the present invention.
8 is a flowchart illustrating a method of generating a storyboard according to an embodiment of the present invention.
FIG. 9 is a flowchart illustrating an operation of a method for producing an image storyboard through an image storyboard maker according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. It should be understood, however, that the present invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, the same reference numbers are used throughout the specification to refer to the same or like parts.

Throughout this specification, when an element is referred to as being "connected" to another element, it is intended to be understood that it is not only "directly connected" but also "electrically connected" or "indirectly connected" "Is included.

It will be appreciated that throughout the specification it will be understood that when a member is located on another member "top", "top", "under", "bottom" But also the case where there is another member between the two members as well as the case where they are in contact with each other.

Throughout this specification, when an element is referred to as "including " an element, it is understood that the element may include other elements as well, without departing from the other elements unless specifically stated otherwise.

Before acquiring real-life aerial images, we are developing 3D aerial images, 3D simulation technology, 3D image technology, and image processing technology, which are suitable for low-cost 3D aerial images And more particularly, to a system and method for generating a storyboard.

1 is a diagram showing a schematic configuration of a storyboard generating system 10 according to an embodiment of the present invention. Prior to the description, the storyboard generating system 10 according to an embodiment of the present invention may be referred to as a dronesimaging support system using aerial integration simulation.

1, a storyboard creating system 10 according to an embodiment of the present invention includes a ground control system 100, a simulation data exchange unit 200, an aviation graphic simulator 300, an image storyboard maker 400, And a database 500.

The ground control system 100 may transmit to the aviation graphic simulator 300 a flight scenario including virtual shooting environment information and shooting mode information related to the UAV.

A description of the terrestrial control system 100 may be more readily understood with reference to FIG.

2 is a flowchart illustrating an operation of the terrestrial control system 100 in the storyboard creating system 10 according to an embodiment of the present invention.

Referring to FIG. 2, the ground control system 100 may set control commands or flight scenarios for controlling the flight simulator and UAV included in the aviation graphic simulator 300 based on the user (or operator) input. Here, the UAV may be a virtual UAV flying in the air graphic simulator 300.

The ground control system 100 may include a user interface (not shown) for receiving a control command or a flying scenario for controlling the flight simulator and the UAV from a user. The user interface may include a keyboard, a touch screen, a mouse, .

The user may access the flight simulator through the simulation data exchange unit 200 to control the flight simulator through the ground control system 100, and then transmit the flight scenario to the aerial graphic simulator 300 (S21). At this time, the UAV can be controlled on the basis of the flight scenario, and the user can monitor, in real time, the information (for example, flight information and the like) related to the UAV and the image photographed by the UAV through the ground control system 100 .

The ground control system 100 may perform a TCP (UDP) / IP connection to the simulation data exchange unit 200 and the simulation data exchange unit 200 may transmit the TCP (UDP) / IP connection to the ground control system 100, And the aviation graphic simulator 300 on the network. Here, examples of the network include wireless sensor networks (WSNs), Bluetooth networks, NFC (Near Field Communication) networks, Internet, LAN (Local Area Network), Wireless LAN Area Network), and the like.

In addition, the flight scenario may be pre-set by a user who wishes to acquire an aerial shot image, such as a movie director or a cinematographer. The predetermined flight scenario information may include virtual photography environment information to be used in the flight simulator and photographing method information related to the UAV.

The virtual photographing environment information is information on a photographing environment required by a user such as a film director or a photographing director, and may include background-related information such as a natural landscape, a building, etc., and information related to an object . The shooting method information related to the UAV includes not only the information of the UAV model, the type of the UAV module, the flight path of the UAV (or the movement line of the UAV, the flight speed etc.), the movement speed of the UAV according to the camera shooting technique, Information about the same camera photographing method, and the like. Particularly, in the case of a UAV flight path, a panning technique for shooting the UAV from left to right or from the right to the left, a tilt technique for shooting the UAV in the vertical and vertical directions, Or a zooming technique in which a subject is photographed while moving away from the subject, or vice versa.

Since the UAV can move various paths as it moves in the three-dimensional space, the ground control system 100 can set the flight scenarios according to various paths in the same virtual photographing environment. In other words, the ground control system 100 can set control command information related to the flight simulator and the UAV as flight scenarios so as to perform repetitive shooting according to various flight paths and camera configurations. Then, the ground control system 100 can transmit the set flight scenario to the aviation graphic simulator 300 through the simulation data exchange unit 200.

After the flight scenario is transmitted (S21), the aviation graphic simulator 300 controls the UAV based on the flight scenario, and can acquire the photographed image by controlling the UAV. The ground control system 100 can then receive the captured image (in other words, the captured image captured through the UAV) from the aviary graphic simulator 300 (S22).

In step S22, the terrestrial control system 100 can receive not only the image captured through the UAV, but also the flight information of the UAV at the time of shooting the captured image and the time information at the time of shooting the corresponding captured image. Here, the flight information of the UAV may include position information of the UAV, latitude, longitude and altitude, GPS information, roll, pitch, and yaw Attitude information, speed information, flight path information, and the like. In addition, in step S22, the ground control system 100 can receive the flight information of the shot image and the UAV according to the delay time set in the air graphic simulator 300, and the description of the delay time will be described in more detail with reference to FIG. 3 .

At this time, the ground control system 100 can receive the photographed image transmitted from the aviation graphic simulator 300 through an image receiving unit (not shown). In the case of the image, since the capacity is large unlike the flight information of the text form, a delay of the entire system 10 may occur in the process of receiving the image from the air graphic simulator 300. Accordingly, the image receiving unit (not shown) may be implemented separately from the terrestrial control system 100, and the image receiving unit (not shown) may be implemented as asynchronous data communication so that the entire system 10 Performance degradation can be minimized.

Next, in step S23, the ground control system 100 can transmit the photographed image and the UAV's flight information received from the aviary graphic simulator 300 to the video storyboard maker 400. FIG. Thereafter, the video storyboard maker 400 can generate a storyboard based on the shot image and the UAV's flight information received from the terrestrial control system 100, which will be described later in more detail.

After step S23, the terrestrial control system 100 can check whether the simulation for the flight scenario transmitted in step S21 is terminated (S24). At this time, if it is determined that the simulation is not ended (S24-N), the ground control system 100 may return to step S22. On the other hand, if it is determined that the simulation for the flight scenario transmitted in step S21 is terminated (S24-Y), the terrestrial control system 100 can confirm whether or not the next flight scenario exists (S25). If it is determined that the next flight scenario exists (S25-Y), the ground control system 100 returns to step S21. If it is determined that the next flight scenario does not exist (S25-N) 100 can be terminated.

The simulation data exchange unit 200 is an interface between the ground control system 100 and the aviation graphic simulator 300 and connects the ground control system 100 and the aviation graphic simulator 300 on the network . In the case of using the commercial flight simulator, it is difficult to directly control the flight simulator in the external software. Therefore, the storyboard generating system 10 according to the embodiment of the present invention is capable of supporting the flight simulator through the simulation data exchanger 200 The flight simulator can be indirectly controlled by using API (Application Program Interface). Accordingly, the simulation data exchange unit 200 calls the API corresponding to the control command in accordance with the control command of the ground control system 100 for driving the flight simulator, so that the ground control system 100 and the air graphic simulator 300 can be connected by a network.

The aviation graphic simulator 300 controls the UAV based on the flight scenario received from the ground control system 200 and can provide the photographed image photographed through the UAV to the ground control system 200. The description of the aviation graphic simulator 300 can be more easily understood with reference to Fig.

3 is a flowchart illustrating an operation flow of the aviation graphic simulator 300 in the storyboard creating system 10 according to an embodiment of the present invention.

Referring to FIG. 3, the aviation graphic simulator 300 may be an apparatus in which a flight simulator and a movie set scene graphic simulator are interlocked. The flight simulator may include, for example, a photographing module, an image storing module, and a data transmitting module (e.g., image, flight information, etc.). The movie set scene graphic simulator may be a device for providing a shooting environment such as background, character, vehicle, etc. of a movie shooting place using computer graphics technology.

In step S31, the aviation graphic simulator 300 can construct a virtual flight environment (or a virtual photography environment) in the flight simulator.

For example, a flight simulator such as an X-plane or a flight simulator has a tool (for example, World Editor) that can add a 3D model such as a background, a building, and an actor to a simulation environment. In step S31, The graphics simulator 300 may use such open or commercial 3D modeling tools to model the environment and objects to use in the flight simulator. In addition, a scenario for the movement of each object (e.g., a person, an automobile, etc.) may be set in the aviation graphic simulator 300.

The flight simulator included in the aviation graphic simulator 300 can perform a TCP (UDP) / IP connection to the simulation data exchange unit 200 after the flight environment is established or when the flight environment is established. The simulation data exchange unit 200 may be interlocked with N flight simulators according to the operating environment. In this case, for example, the simulation data exchange unit 200 may be a server type and the flight simulator may be a client type network.

After connecting to the simulation data exchange unit 200, the aviation graphic simulator 300 can receive the flight scenario from the ground control system 100. [ The aviation graphic simulator 300 can confirm (or initialize) the simulation environment according to the flight scenario if it is determined that the flight scenario has been received in step S32 after confirming whether the flight scenario has been received (step S32). Through this process, the aviation graphic simulator 300 can set the virtual shooting environment and the UAV to be simulated before performing the simulation.

Thereafter, the aviation graphic simulator 300 controls the UAV based on the received flight scenario (S33), and acquires the captured image through the UAV (S34) by controlling the UAV.

More specifically, after receiving the flight scenario, the aviation graphic simulator 300 can execute the simulation of the flight scenario by driving the UAV in the established virtual shooting environment and performing image shooting through UAV. In this case, the UAV can perform the image shooting while flying along the predetermined route based on the flight scenario, and the UAV can view the object in the air when shooting through the UAV. For example, the UAV may be an extreme long shot, a long shot, A full shot, a loose shot, and the like can be captured and acquired.

In addition, the aviation graphic simulator 300 can acquire the photographed image by using the zoom-in and zoom-out functions of the flight simulator itself as well as the photographed image through the UAV. The photographed images photographed using the zoom-in / zoom-out function of the flight simulator itself may include images such as nose shot, waist shot, bust shot, close shot, close-up shot, extreme close-up shot, and tight shot. Accordingly, when the aerial graphic simulator 300 transmits the photographed image to the ground control system 100 in step S35, which will be described later, the aerial graphic simulator 300 may transmit the photographed image photographed through the UAV, .

That is, in the storyboard generating system 10 according to the embodiment of the present invention, the ground control system 100 obtains only the photographed image photographed through the UAV, as the photographed image acquired from the aviation graphic simulator 300 However, the present invention is not limited thereto. As described above, the aerial graphic simulator 300 may include an aerial image simulated by the aerial graphic simulator 300. Accordingly, the video storyboard maker 400, which will be described later, not only generates the storyboard based on the shot image photographed through the UAV, but also generates the storyboard based on the shot image photographed through the shooting module built in the aviation graphic simulator 300 You can also create a storyboard. Alternatively, the video storyboard maker 400 may generate a storyboard using an image captured through the UAV and an image captured through the self-contained shooting module of the air graphic simulator 300 .

Meanwhile, in step S34, the aviation graphic simulator 300 can acquire the photographed image in consideration of the predetermined delay time. Here, the delay time may mean a waiting time for transmitting the photographed image photographed through the UAV (or the photographed image photographed through the photographic module provided in the aviation graphic simulator 300 itself) to the ground control system 100 And the image acquisition speed at which the aviation graphic simulator 300 acquires the photographed image photographed through the UAV can be determined by the predetermined delay time. For example, when the delay time is set to 0.1 second, the aviation graphic simulator 300 can acquire the photographed image photographed through the UAV at a rate of up to 10 frames / second.

In addition, the aviation graphic simulator 300 may capture an image photographed through a UAV, for example, and store the captured image in a database 500 to be described later. At this time, the aviation graphic simulator 300 may store the flight information of the UAV when capturing the captured image in the database 500 together with the captured image at the time of storing the captured image.

After step S34, the aviation graphic simulator 300 can transmit the shot image shot through the UAV and the flight information of the UAV when shooting the shot image together to the ground control system 100 (S35).

After step S35, the aviation graphic simulator 300 can confirm whether the simulation for the flight scenario received from the terrestrial control system 100 has been completed (S36). If it is determined that the simulation for the flight scenario is not terminated (S36-N), the aviation graphic simulator 300 returns to step S34 to acquire the shot image photographed through the unmanned aerial vehicle, and the simulation for the flight scenario is terminated If it is judged (S36-Y), the operation of the aviation graphic simulator 300 can be terminated.

On the other hand, the database 500 may store all information related to the storyboard creation system 10. For example, the database 500 may include information on flight scenarios set through the ground control system 100, an image captured through an aerial graphic simulator 300 (e.g., an image captured through an unmanned aerial vehicle, The flight information of the UAV, the storyboard image selected through the image storyboard maker 400, and the storyboard generated through the image storyboard maker 400. [0051] FIG.

The video storyboard maker 400 acquires a shot image from the ground control system 100, selects a storyboard image based on a preset situation condition from the acquired shot image, Can be generated.

The description of the video storyboard maker 400 can be more easily understood from FIG. 4 through FIG.

4 is a block diagram schematically showing a configuration of an image storyboard maker 400 according to an embodiment of the present invention.

Referring to FIG. 4, the video storyboard maker 400 according to one embodiment of the present invention may include a situation setting unit 410, a video analyzing unit 420, and a storyboard generating unit 430.

The video storyboard maker 400 can receive input from a user from a user a control input related to creating a storyboard (for example, setting a condition condition, setting a maximum number of storyboard images, setting an algorithm, etc.) (Not shown) included in the terrestrial control system 100 or a user included in the video storyboard maker 400 separately from the user interface included in the terrestrial control system 100, And input via an interface (not shown). The user interface included in the video storyboard maker 400 may also be implemented by a keyboard, a touch screen, a mouse, and the like, but is not limited thereto.

The situation setting unit 410 can set a situation condition corresponding to a characteristic for selecting a scene stored in the storyboard. In other words, the situation setting unit 410 may set a reference condition for extracting a scene to be selected as a storyboard image from scenes included in the photographed image, and the reference condition may be a situation condition or a storyboard image Extraction conditions, and the like. Here, a scene may mean a cut corresponding to one frame, or a set of a plurality of frames that can be recognized as a single situation. For example, a scene can be understood to be the same as or similar to the concept of a scene containing one scene. In addition, the situation condition may include a situation variable that is given in accordance with the importance of the characteristic appearing in each scene included in the shot image in the selection of the storyboard image. The context variables can be, for example, variables that quantify features or events that can be identified through analysis algorithms such as the number of people, the number of vehicles, a specific action of a person, a specific event, and the like.

More specifically, the context condition may include an object context variable for an object included in the captured image, and the object context variable may be a variable related to at least one of an object's property, a number of objects, and a location of the object.

For example, a variable related to the characteristics of an object is a variable depending on the kind or characteristics of an object, and it is a variable such as a person or object, a man or woman, a person, a star or an assistant, Whether it is a vehicle, a plane, a train, a train, a bus, or a natural property, such as a natural building, a building or a tree, etc. If the vehicle is a vehicle A or a vehicle B, Color, and B color, and the like, it may include variables considering the apartment, the low-rise house, the high-rise building, or the tower-type apartment in the case of the building, and the characteristics of the face, the body, And the characteristics of objects. Variables related to the number of objects can include variables that take into account whether the number of objects is 0, 1 or 2. Variables related to the position of an object include variables related to the position, such as whether the object is in a stopped state or moving state, in which direction it is moved, on which path it travels, .

In addition, the conditional condition may include a screen condition variable for the screen change of the photographed image, and the screen condition parameter may be a variable related to at least one of whether the background is switched, whether the background is rapidly changed, and the background switching speed. Here, the background may include a background such as a sky, a background in an indoor space, a background in an outdoor space such as a road, an alley, a boulevard, and the like.

In addition, the background may include objects that are not assigned importance. For example, a specific scene included in a photographed image is a scene including a passenger car parked at a bus stop in front of a building and a person standing next to the passenger car. In this scene, . At this time, generally, each building, passenger car, and person can be recognized as an object situation variable in a specific scene, but in the above assumption, it is assumed that only a person is a situation variable to which importance is given, , Passenger cars and buildings can be recognized as backgrounds even though they are classified as objects due to their properties but they are not assigned importance. Accordingly, when the object such as an airplane is recognized as a background, variables related to the change of the flight path of the airplane may be included in the screen status variable.

In addition, the situation setting unit 410 may set an analysis algorithm for selecting a scene stored in the storyboard among the scenes included in the shot image. The situation setting unit 410 can be more easily understood with reference to Fig.

5 is a diagram illustrating an example of the operation flow of the situation setting unit 410 in the video storyboard maker 400 according to an embodiment of the present invention.

Referring to FIG. 5, in step S51, the situation setting unit 410 may set a maximum number of storyboard images included in a storyboard based on input of a user (for example, a movie director, a cinematographer, etc.). At this time, the maximum number of storyboard images can be set by the user in consideration of the story flow and the total length (or time) of the shot images.

In this case, if the maximum number of storyboard images is set to a small value, the accuracy may be lowered when photographing images through an actual UAV. If the maximum number of storyboard images is set to a large value, If you want to use it as a reference for shooting, it may take a lot of time to review the storyboard. Therefore, the maximum number of storyboard images is set to an appropriate number considering the development of the story and the shooting motion of the UAV. desirable.

On the other hand, in the case of the storyboard generated through the storyboard generating unit 430 to be described later, not only the shot image corresponding to the storyboard image but also the storyboard image corresponding to the storyboard image And the flight information of the UAV of the UAV can be included. Thus, each storyboard image included in the storyboard can be used as a way point of the UAV when the image is captured through the actual UAV. In addition, each storyboard image included in the storyboard can be used as an editing point by the cinematographer or film director.

Next, in step S52, the situation setting unit 410 can set an algorithm to be used by the video analysis unit 420 to select scenes stored in the storyboard among the scenes included in the shot image. In other words, the situation setting unit 410 can set an algorithm for identifying a scene corresponding to the set conditional condition.

For example, the algorithms that can be set by the situation setting unit 410 include a face detection algorithm, a multiple face detection algorithm, a gender detection algorithm, a human body detection algorithm, an object (such as a person, a vehicle, a train, ) Detecting algorithm, a screen switching algorithm, a moving direction detecting algorithm, and a detection algorithm capable of identifying a specific situation of a photographed image, and the like, but the present invention is not limited thereto.

Based on a plurality of algorithms previously set in the situation setting unit 410, a video analyzing unit 420, which will be described later, calculates a scalar value for each scene included in the photographed image by using an algorithm corresponding to each situation variable included in the situation condition Value can be calculated, and a storyboard image can be selected through this. In addition, the video analysis unit 420 may select a storyboard image by using a plurality of algorithms corresponding to each of a plurality of redeeming variables, when the situation condition includes a plurality of situation variables. That is, in selecting the storyboard image, the video analyzer 420 may select a storyboard image using one algorithm or a combination of two or more algorithms. In addition, the video analysis unit 420 can select a storyboard image by considering a combination of priorities or posterior relations among analysis algorithms. At this time, the priority among the analysis algorithms can be determined based on the degree of importance of each property given according to the situation variable when setting the condition condition. For example, in a scene involving a woman and a tree, if a woman is given a higher importance than a tree, an algorithm for identifying a person may be used first rather than an algorithm for identifying the tree.

Next, in step S53, the situation setting unit 410 may configure the video analysis logic. In more detail, the situation setting unit 410 may be configured to set a storyboard image based on a maximum number of predetermined storyboard images and a predetermined analysis algorithm, You can configure the video analysis logic to change the variables.

On the other hand, the video analyzer 420 can select a storyboard image from the photographed image based on the status condition set in the situation setting unit 410. [ At this time, the photographed image may be an image photographed through a UAV, where the UAV may mean a virtual UAV flying in the aviation graphic simulator 300. As described above, the photographed image may include not only the image photographed through the UAV, but also the image photographed through the photographing module provided in the aviation graphic simulator 300 itself.

In addition, the video analyzer 420 can extract a scene that satisfies the set condition set by the situation setting unit 410 in the shot image, and can select the extracted scene as the storyboard image.

In addition, when a plurality of context variables are included in the context condition, the video analysis unit 420 may select a storyboard image by considering a plurality of context variables together. At this time, the plurality of context variables may be expressed as scalar values, and the video analyzer 420 may calculate the storyboard image based on the sum information obtained by summing the scalar values of the plurality of context variables for each scene included in the shot image Can be selected.

In addition, the video analyzer 420 sorts the sum information values calculated for each of the scenes included in the photographed image in order of magnitude (for example, in descending order of magnitude), and then, based on the summed sum information values, Scenes corresponding to the maximum number of storyboard images preset in the unit 410 can be selected as storyboard images.

If the sum information values of the frames from the nth frame to the (n + 10) th frame are all the same among the frames included in the photographed image, the frame is different from the nth frame to the (n + 10) th frame Can be recognized as a scene having a single meaning, that is, as a single scene as the same situation. That is, when the same summation information value is calculated for neighboring frames on the timeline of the photographed image, it can be recognized that neighboring frames having the same summation information value are the same scene. In this case, the video analyzer 420 may combine one frame or a plurality of frames among neighboring frames having the same sum information value (for example, frames from the nth frame to the (n + 10) One frame may be extracted as a frame representative of the scene, and the extracted frame may be selected as a storyboard image, but the present invention is not limited thereto.

Meanwhile, for example, when a scene including three persons and one vehicle among the scenes included in the photographed image is to be selected as a storyboard image, the situation setting unit 410 sets three conditions One condition condition may be set. At this time, in order to select, as a storyboard image, a scene in which three persons are included in the shot image and one vehicle is included among the scenes included in the shot image, the video analyzer 420 considers the number of human objects A situation variable can be used that takes into account the context variables and the number of objects that are objects (ie, vehicles), and a scalar value for each context variable (or a weight value as the importance assigned to each context variable) can be calculated.

More specifically, for example, a scene including three persons among the scenes included in the photographed image has a weight of 10 points (in this case, the weight is a degree of importance given to each situation variable, and 0 to 10 points , A scene having two or four persons is given a weight of 7, and a scene having one or five persons is given a weight of 3 points And a scene having 0 or 6 or more persons may be given a weight of zero. Of the scenes included in the photographed image, for example, a scene including one vehicle is given a weight of 10 points, and a scene including zero vehicles (i.e., a scene without a vehicle) A weight is given, and a scene having two or more vehicles can be given a weight of zero.

In the situation setting unit 410, a situation condition in which one person comes down from the passenger car in addition to a situation condition in which there are three persons and one vehicle, a situation condition in which five persons move from a point to point b, A situation condition may be set. At this time, the video analyzer 420 calculates the sum information of the scalar values for the context variables included in each of the contextual conditions for each of the plurality of contextual conditions set by the context setting unit 410, for each scene included in the captured image can do.

For example, as the summation information, the video analyzing unit 420 calculates a scalar value corresponding to a weight of 20 points for a scene including three persons and one vehicle, and includes three persons and zero vehicles A scalar value corresponding to a weight of 13 points can be calculated for a scene in which a person is present and a scalar value corresponding to a weight of three points can be calculated for a scene in which one person and two vehicles are included. For example, the video analyzing unit 420 may calculate a scalar value corresponding to a weight of 30 points for a scene in which five people move from a point to point b as summing information, , A scalar value corresponding to the weight of 14 points is calculated, and when one person moves from point a to point c opposite to point b, a scalar corresponding to the weight of two points can be calculated.

As described above, the video analyzer 420 can calculate the scalar value based on not only the status variables related to the number of objects such as the number of people, the number of vehicles, Scalar values can be calculated using context variables that can be identified (or recognized) for the situation.

Then, the video analyzer 420 sorts the sum information values according to the calculated scalar values for each of the scenes included in the photographed image in order of magnitude (for example, in descending order of size) Scenes corresponding to the maximum number of preset storyboard images may be selected as storyboard images. At this time, the upper list among the scenes sorted in order of magnitude may include a plurality of scene conditions that are most similar to or more similar to the plurality of situation conditions set in the situation setting unit 410, The scene that best matches the situation condition may be located. Accordingly, the video analyzer 420 can select, as a storyboard image, a scene corresponding to the maximum number of storyboard images on the basis of the order in which the sum information value is high.

In addition, the video analysis unit 420 can select a storyboard image by analyzing the shot image photographed through the UAV, and sort the selected storyboard images in chronological order, which can be more easily understood from FIG. 6 .

Referring to FIG. 6, in step S61, the video analysis unit 420 may analyze the photographed image to select a storyboard image. To this end, the video analysis unit 420 can acquire the photographed image photographed through the UAV from the terrestrial control system 100 first. At this time, the obtained captured image may be an image photographed through a UAV, or an image photographed through a photographing module provided in the air graphic simulator 300 itself.

In addition, the photographed image obtained from the terrestrial control system 100 may be a moving image format or a still image format, and the video analysis unit 420 may convert the format into a format that can be analyzed according to the image format. For example, when the photographed image obtained from the terrestrial control system 100 is a video file format of the AVI format, the video analysis unit 420 can convert it into the RGB format of the Mat structure using the OpenCV library. The video analyzer 420 may then pre-process the input image, which has been converted into an analytical format, to facilitate analysis.

Then, the video analysis unit 420 selects a storyboard image (S62) by performing an image analysis for detecting a specific situation, a specific person, and the like on the photographed image based on the set conditional condition set by the situation setting unit 410, can do.

The video analysis unit 420 can select a scene corresponding to a specific situation in the scenes included in the shot image as a storyboard image by using the algorithm defined in the situation setting unit 410. [ For example, in order to select a scene corresponding to the 'A' situation in the photographed image as a storyboard image, the video analyzer 420 may combine a, b, and c algorithms for detecting the 'A' To perform image analysis. At this time, the video analyzer 420 may sort the calculated scalar values for each scene included in the photographed image through image analysis, and select a storyboard image based on the sorted scalar value.

Then, the video analysis unit 420 may store the selected storyboard image in the database 500. [

On the other hand, the storyboard generating unit 430 combines the storyboard image selected by the video analyzer 420 and the flight information of the UAV at the photographing point corresponding to the selected storyboard image to generate a storyboard . At this time, the flight information of the UAV at the time of photographing corresponding to the storyboard image can be used as a way point of the actual UAV when the image is captured through the actual UAV.

In addition, the flight information of the UAV may include UAV information about the shooting time corresponding to the storyboard image in the captured image, and the storyboard generating unit 430 may transmit UAV time information to the system time information , It is possible to correct the error of the UAV time information based on the system time information when it is determined that there is an error in the UAV time information due to the delay caused by the temporary failure of the UAV, To create a storyboard.

The description of the storyboard generating unit 430 can be more easily understood with reference to FIG.

7 is a flowchart illustrating an operation flow of the storyboard generating unit 430 in the video storyboard maker 400 according to an embodiment of the present invention.

Referring to FIG. 7, the storyboard generation unit 430 generates a storyboard image by adding the storyboard images selected by the image analysis in the video analysis unit 420, as additional information, After the information is integrated, the storyboard images considering the flight information of the UAV can be sorted in chronological order and generated as one storyboard. At this time, as the flight information of the UAV, the storyboard generating unit 430 may include not only attitude information of the UAV, position information, time information of the image corresponding to the storyboard image, Lt; / RTI > Accordingly, the storyboard generating unit 430 can generate the storyboard by including the flight information of the UAV, which is available when shooting the image through the actual UAV, together with not only the storyboard by using only the captured image , And the storyboard generated through the storyboard generating unit 430 can be utilized as photographing indicators useful in real-time photographing through UAV.

For this purpose, in step S71, the storyboard generating unit 430 can inquire the photographing time of the storyboard image.

At this time, the shooting time of the storyboard image may be time information of the shot image corresponding to the storyboard image taken by the UAV.

On the other hand, when the photographed image photographed by the UAV is stored in the storyboard creating system 10, not only the time information (i.e., UAV information) captured by the UAV, but also the photographed image is recorded in the storyboard creating system 10 (I.e., system time information) may be stored together with the time information. For example, when a delay occurs in the image acquisition time through the UAV due to a temporary failure during the simulation through the aviation graphic simulator 300 and thus an error occurs in the UAV time information, As shown in FIG.

Accordingly, the shooting time of the storyboard image may be the time information of the shot image corresponding to the storyboard image taken by the UAV, and may be the time information whose error has been corrected by the system time information. The time information between the storyboard images can be used to determine the thrust, attitude control, and the like of the UAV according to the external environment when photographing the image through the actual UAV. In addition, it is possible to estimate the broadcast time of the video image during aerial image capturing through the real UAV through time information between storyboard images.

Next, in step S72, the storyboard generating unit 430 can inquire about the flight information of the UAV at the photographing time corresponding to the storyboard image. At this time, the flight information of the UAV may include all information related to the flight of the UAV, such as position information such as latitude, longitude, and altitude, attitude information such as roll, pitch, yaw, speed information and the like.

Then, in step S73, the storyboard generating unit 430 generates a storyboard by integrating the searched UAV time information and the UAV information on each of the storyboard images selected by the video analyzer 420, can do.

Each of the storyboard images included in the generated storyboard is added to the index of the shot image so that the flight information and the steering information of the same index can be shared. Accordingly, for example, when a movie director or the like selects the storyboard, the film director can receive flight information and maneuvering information of the UAV by using the index values included in the storyboard. When the movie director inputs the flight information or the control information value of the UAV provided through the storyboard as a control input value of the actual UAV, the actual UAV is inputted to the designated flight path based on the inputted UAV's flight information or the control information value It is possible to take an image through a camera or the like mounted on a real UAV while driving along. At this time, the actual UAV can shoot the same or similar live image as the simulation image, and the captured live image can be stored in the storage module provided in the actual UAV, or transmitted to the ground control system 100. For example, the terrestrial control system 100 may store a real image obtained through an actual UAV in a real image database (not shown), and the image stored in a real image database (not shown) .

In addition, the storyboard generating system 10 according to an embodiment of the present invention may use a flight simulator or a virtual unmanned aerial vehicle to perform composition of a camera, a moving line of a character (character), an atmosphere of a scene And generates a storyboard based on the acquired image. At this time, in generating the storyboard, the video storyboard maker 400 generates a storyboard creation condition (for example, a variety of options for selecting a storyboard video) preset by a user, such as a movie director or a cinematographer, Condition setting, and so on). Accordingly, the user can quickly review a plurality of photographed images photographed in various compositions according to various flight scenarios based on a storyboard generated for each of the plurality of photographed images. In addition, since the storyboard includes not only shot images but also flight information of UAV, shooting of the scenes can be easily performed on the basis of the storyboard when shooting actual aerial images. Accordingly, the present invention can effectively reduce the time and cost required for aerial image capturing, and prevent accidents caused by repetitive operations of aircraft and the like.

Hereinafter, the operation flow of the present invention will be briefly described based on the details described above.

8 is a flowchart illustrating a method of generating a storyboard according to an embodiment of the present invention.

The storyboard generating method shown in FIG. 8 can be performed by the storyboard generating system 10 described above. Therefore, even if the contents are omitted in the following description, the contents described with respect to the storyboard creating system 10 can be similarly applied to FIG.

Referring to FIG. 8, in step 810, the ground control system 100 may transmit to the aviation graphic simulator 300 a flight scenario including virtual photography environment information and photographing method information related to the UAV.

Next, in step S820, the aviation graphic simulator 300 controls the UAV based on the flight scenario received from the ground control system 100, and provides the captured image through the UAV to the ground control system 100 have.

Next, in step S830, the video storyboard maker 400 acquires the shot image from the terrestrial control system 100, and uses the selected storyboard image based on the preset situation condition from the shot image to obtain a flight scenario Can be generated.

In the above description, steps S810 through S830 may be further divided into further steps, or combined in fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

9 is a flowchart illustrating a method for producing an image storyboard through a video storyboard maker according to an embodiment of the present invention.

9 may be performed by the video storyboard maker 400 described above. Accordingly, the contents described with respect to the video storyboard maker 400 can be similarly applied to FIG. 9, even if omitted from the following description.

Referring to FIG. 9, the situation setting unit 410 of the video storyboard maker 400 may set a condition condition corresponding to a characteristic for selecting a scene stored in the storyboard (S910).

Next, the video analyzer 420 of the video storyboard maker 400 may select a storyboard image from the shot image based on the set conditional condition at step S910 (S920).

Next, the storyboard generating unit 420 of the video storyboard maker 400 integrates the flight information of the UAV at the shooting point corresponding to the storyboard image in the storyboard image and the shot image selected in step S920, (S930).

In the above description, steps S910 through S930 may be further divided into further steps, or combined in fewer steps, according to embodiments of the present disclosure. Also, some of the steps may be omitted as necessary, and the order between the steps may be changed.

The storyboard creating method and the story story board creating method according to an exemplary embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described storyboard generating method and image storyboard producing method may be implemented in the form of a computer program or an application executed by a computer stored in a recording medium.

It will be understood by those of ordinary skill in the art that the foregoing description of the embodiments is for illustrative purposes and that those skilled in the art can easily modify the invention without departing from the spirit or essential characteristics thereof. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be interpreted as being included in the scope of the present invention.

10: storyboard production system
100: Ground control system
200: Simulation data exchange part
300: Aviation graphic simulator
400: Video storyboard maker
500: Database

Claims (19)

A ground control system for transmitting a flight scenario including virtual shooting environment information and shooting mode information related to a UAV to an aerial graphic simulator;
An aviation graphic simulator for controlling the unmanned aircraft based on the flight scenario received from the terrestrial control system and providing the photographed image photographed through the unmanned aerial vehicle to the terrestrial control system; And
An image storyboard maker for acquiring the photographed image from the terrestrial control system and generating a storyboard corresponding to the flight scenario using the selected storyboard image based on a predetermined situation condition from the obtained photographed image,
, ≪ / RTI &
The UAV is a virtual UAV flying in the air graphic simulator,
The video storyboard maker includes:
A situation condition including a situation variable that is given according to the importance of a characteristic appearing in each scene included in the photographed image is set in correspondence with a characteristic for selecting a scene to be stored in the storyboard A situation setting section;
A video analysis unit for selecting a storyboard image from the photographed image based on the situation condition, and
And a storyboard generating unit for generating the storyboard by integrating the storyboard image and the flight information of the UAV at an imaging time corresponding to each storyboard image in the captured image,
Wherein the context condition includes an object context variable for an object included in the captured image and a screen context variable for screen transition of the captured image,
The flight information of the unmanned aerial vehicle at the shooting time corresponding to each of the storyboard images includes position information, attitude information, and speed information of the unmanned aerial vehicle. Wherein the storyboard creation system is used to create a storyboard. delete delete delete The method according to claim 1,
Wherein the flight information of the UAV includes UAV time information about a shooting time point corresponding to the storyboard image in the captured image,
The storyboard generating unit compares the UAV time information with the system time information of the system controlling the UAV, and when it is determined that there is an error in the UAV time information, And correcting the error. delete delete The method according to claim 1,
Wherein the object context variable is a variable associated with at least one of an object's property, a number of objects, and an object's location. delete The method according to claim 1,
Wherein the screen status variable is a variable related to at least one of whether or not the background is switched and the switching speed of the background. The method according to claim 1,
Wherein the video analysis unit selects the storyboard image by considering the plurality of context variables together when a plurality of context variables are included in the context condition. 12. The method of claim 11,
Wherein the plurality of context variables are scalar values,
Wherein the video analysis unit selects the storyboard image based on summation information obtained by summing a plurality of scalar values of the plurality of context variables for each scene included in the captured image. 13. The method of claim 12,
The maximum number of the storyboard images included in the storyboard is set in the situation setting unit,
The video analyzer may sort the summation information values calculated for each of the scenes included in the shot image in order of magnitude and then select scenes corresponding to the maximum number as the storyboard images based on the sorted summation information value Storyboard creation system. An image storyboard maker for generating a storyboard corresponding to a flight scenario using a selected storyboard image based on predetermined conditional conditions from a shot image obtained by shooting through a virtual unmanned aerial vehicle flying in an aviation graphic simulator,
A situation setting unit for setting a situation condition corresponding to a characteristic for selecting a scene stored in the storyboard;
A video analyzer for selecting a storyboard image from the photographed image based on the conditional condition; And
And a storyboard generating unit for generating the storyboard by integrating the storyboard image and the flight information of the UAV at an imaging point corresponding to each storyboard image in the captured image,
Wherein the context condition is a context parameter assigned according to a degree of importance of a characteristic appearing in each scene included in the captured image when the storyboard image is selected as an object situation variable for an object included in the captured image, The screen status variable for screen switching of the screen,
The flight information of the unmanned aerial vehicle at the shooting time corresponding to each of the storyboard images includes position information, attitude information, and speed information of the unmanned aerial vehicle. A video storyboarder. delete delete Transmitting, in the ground control system, a flight scenario including virtual shooting environment information and shooting mode information related to the UAV to an aerial graphic simulator;
Controlling the UAV based on the flight scenario received from the ground control system in the air graphic simulator and providing the captured image photographed through the UAV to the ground control system; And
In the video storyboard maker, the shot image is acquired from the ground control system, and a storyboard corresponding to the flight scenario is generated from the shot image using the selected storyboard image based on predetermined conditional conditions step,
, ≪ / RTI &
The UAV is a virtual UAV flying in the air graphic simulator,
The step of generating the storyboard may include:
A situation condition including a situation variable, which is given according to the importance of a characteristic appearing in each scene included in the shot image, is set in correspondence with a characteristic for selecting a scene stored in the storyboard ;
Selecting a storyboard image from the photographed image based on the conditional condition; And
And generating a storyboard by integrating the flight information of the UAV at the photographing point corresponding to each of the storyboard images in the storyboard image and the shot image,
Wherein the context condition includes an object context variable for an object included in the captured image and a screen context variable for screen transition of the captured image,
The flight information of the unmanned aerial vehicle at the shooting time corresponding to each of the storyboard images includes position information, attitude information, and speed information of the unmanned aerial vehicle. ≪ / RTI > Through a video storyboard maker that generates storyboards corresponding to flight scenarios using selected storyboard images based on predefined situation conditions from captured images obtained by shooting through virtual unmanned aerials flying in an aviation graphic simulator A method for producing a video storyboard,
Setting a situation condition corresponding to a characteristic for selecting a scene stored in a storyboard;
Selecting a storyboard image from the photographed image based on the conditional condition; And
And generating storyboards by integrating the flight information of the UAV at the shooting time corresponding to each of the storyboard images in the storyboard image and the shot image,
Wherein the context condition is a context parameter assigned according to a degree of importance of a characteristic appearing in each scene included in the captured image when the storyboard image is selected as an object situation variable for an object included in the captured image, The screen status variable for screen switching of the screen,
The flight information of the unmanned aerial vehicle at the shooting time corresponding to each of the storyboard images includes position information, attitude information, and speed information of the unmanned aerial vehicle. The method comprising the steps of: A computer-readable recording medium recording a program for executing the method of claim 17 or 18 in a computer.

Images