KR102650406B1 - Apparatus and method for providing video service for monitoring distant scene - Google Patents

Abstract

An apparatus and method for providing an image service for monitoring a distant scene are disclosed, and a method for providing an image service for monitoring a distant scene according to an embodiment of the present application includes (a) setting the target area to a plurality of different angles of view; Acquiring a plurality of original images correspondingly taken at a predetermined point in time, (b) stitching the plurality of original images obtained for each of the plurality of view angle settings to generate a plurality of panoramic images, (c) Setting one panoramic image among a plurality of panoramic images as a reference panoramic image and matching the plurality of panoramic images based on the reference panoramic image, (d) linking viewing settings for monitoring of the target area It may include receiving a user input and (e) outputting a viewport based on a panoramic image corresponding to the user input among the plurality of registered panoramic images.

Description

Apparatus and method for providing video services for monitoring distant scenes {APPARATUS AND METHOD FOR PROVIDING VIDEO SERVICE FOR MONITORING DISTANT SCENE}

This application relates to an apparatus and method for providing video services for monitoring distant scenes. For example, this application is about an image shooting, synthesis, and service method for displaying images captured using camera arrays of various angles of view at different viewpoints for the same distant scene to suit the user's needs.

Recently, with the development of video processing and Internet technology, services that stream high-definition video in real time from remote locations have become common. In addition, research and technology development have been conducted to compress high-resolution images such as 360-degree videos and create viewports considering the user's viewing direction.

Meanwhile, recently, there have been several attempts to promote tourist attractions through the Internet, and these also provide services mainly using images of pre-recorded videos of the tourist attractions based on the Internet.

Most of these high-definition video services are filmed and edited according to a pre-planned scenario or shooting direction, such as dramas or movies. When using 360-degree video, viewers can select the viewpoint, but services that allow users to zoom in on areas of interest or view tourist attractions with a wide angle of view are not yet available. In other words, most of the conventional video services have restrictions that require users to watch the video as is at the time the video producer filmed the scene.

In addition, most tourist services at observatories or migratory bird habitats are conducted in the form of observing distant objects with the naked eye or using a telescope from a specific safe location. In this case, distant objects can be observed with the naked eye at a standard angle of view, and You can also relatively magnify an area or object and observe it with a narrow angle of view.

However, when monitoring a remote target in a specific location, the monitoring quality is variable depending on the weather or time. To solve this problem, methods and devices are being designed to store pre-stored images in telescopes at tourist attractions and to selectively view live or stored images. However, there is no research on techniques for generating and storing distant target images based on various angles of view. Development is insufficient.

The technology behind this application is disclosed in Korean Patent Publication No. 10-1432123.

The purpose of this application is to solve the problems of the prior art described above. Based on a panoramic image stitched after shooting a distant target area in response to a plurality of different view angle settings, the user wants to view the view angle setting, viewpoint, and viewing angle. The purpose is to provide an apparatus and method for providing video services for monitoring distant scenes by extracting and outputting a viewport corresponding to the location.

The purpose of this application is to solve the problems of the prior art described above. A method of photographing a remote monitoring object at a specific location using camera arrays with different angles of view and compositing them on a reference image plane, and a method of photographing and compositing objects at different times The purpose is to provide a method and system for streaming images by extracting a viewport according to the user's requested time, viewpoint (viewing direction), and angle of view.

However, the technical challenges sought to be achieved by the embodiments of the present application are not limited to the technical challenges described above, and other technical challenges may exist.

As a technical means for achieving the above-described technical problem, a method of providing an image service for monitoring a distant scene according to an embodiment of the present application includes (a) viewing the target area at a predetermined viewpoint in response to a plurality of different viewing angle settings; Obtaining a plurality of original images captured in, (b) stitching the plurality of original images obtained for each of the plurality of view angle settings to generate a plurality of panoramic images, (c) Among the plurality of panoramic images Setting one panoramic image as a reference panoramic image and matching the plurality of panoramic images based on the reference panoramic image, (d) receiving user input associated with viewing settings for monitoring of the target area and (e) outputting a viewport based on a panoramic image corresponding to the user input among the plurality of registered panoramic images.

Additionally, in step (c), the reference panoramic image may be an image stitched from the plurality of original images obtained corresponding to a relatively wide angle of view among the plurality of view angle settings.

Additionally, in step (a), the plurality of original images may be received from a plurality of camera modules that each photograph a plurality of division areas in which the target area is divided according to the angle of view setting.

In addition, step (a) may receive the plurality of original images in which a predetermined camera module sequentially photographs a plurality of division areas in which the target area is divided according to the angle of view setting.

Additionally, in step (a), at least one original image corresponding to a preset region of interest among the target regions may be acquired.

In addition, the plurality of view angle settings include a standard view angle setting preset for the target area, a wide angle setting for shooting a wide area compared to the area taken by the standard view angle setting, and a comparison of the area taken by the standard view angle setting. It may include a telephoto setting for shooting narrow areas.

Additionally, in step (d), the user input may include the user's viewing position information, viewing direction information, shooting viewpoint information, and view angle setting information for the target area.

In addition, in step (e), the viewing location information is obtained from a panoramic image generated by being captured at a predetermined viewpoint corresponding to the capturing viewpoint information according to a viewing angle setting corresponding to the viewing angle setting information among the plurality of registered panoramic images. And the viewport corresponding to the viewing direction information may be partially extracted and displayed.

In addition, if the panoramic image generated according to the angle of view setting corresponding to the angle of view setting information is an image generated based on at least one original image corresponding to a preset area of interest among the target areas, step (e) may display the viewport for the region of interest that is relatively close to the viewing location information and the viewing direction information.

Meanwhile, an image service providing device for monitoring a distant scene according to an embodiment of the present application includes an image collection unit that acquires a plurality of original images taken at a certain point in time corresponding to a plurality of different view angle settings of the target area. , a stitching unit for generating a plurality of panoramic images by stitching the plurality of original images obtained for each of the plurality of view angle settings, setting one panoramic image among the plurality of panoramic images as a reference panoramic image, and A matching unit that matches the plurality of panoramic images based on the image and receives user input associated with viewing settings for monitoring the target area, and selects a panoramic image corresponding to the user input among the plurality of registered panoramic images. It may include an image playback unit that outputs a viewport based on .

In addition, the image collection unit receives the plurality of original images from a plurality of camera modules that each photograph a plurality of division areas dividing the target area according to the angle of view setting, or divides the target area according to the angle of view setting. The plurality of original images sequentially photographed by a predetermined camera module in a plurality of segmented areas may be received.

Additionally, the image collection unit may acquire at least one original image corresponding to a preset area of interest among the target areas.

In addition, the video reproducing unit may display the viewing position information and the The viewport corresponding to viewing direction information may be partially extracted and displayed.

In addition, if the panoramic image generated according to the angle of view setting corresponding to the angle of view setting information is an image generated based on the at least one original image corresponding to a preset area of interest among the target areas, the image reproduction unit, The viewport for the area of interest that is relatively close to the viewing location information and the viewing direction information may be displayed.

The above-described means of solving the problem are merely illustrative and should not be construed as intended to limit the present application. In addition to the exemplary embodiments described above, additional embodiments may be present in the drawings and detailed description of the invention.

According to the above-described means of solving the problem of this application, based on a panoramic image stitched after shooting a distant target area in response to a plurality of different angles of view, An image service providing device and method for monitoring a distant scene that is output by extracting a viewport can be provided.

According to the above-described means of solving the problem of the present application, a method of photographing a remote monitoring object at a specific location using camera arrays of different angles of view and compositing them on a reference image plane is used, and the images photographed and synthesized at different times are used by the user. A method and system for extracting and streaming a viewport according to the request time, viewing direction, and angle of view can be provided.

However, the effects that can be obtained herein are not limited to the effects described above, and other effects may exist.

1 is a schematic configuration diagram of a remote video service providing system including a video service providing device for monitoring a distant scene according to an embodiment of the present application.
Figure 2 is a conceptual diagram showing a method of capturing an original image of a target area using a camera array with a standard angle of view setting.
Figure 3 is a conceptual diagram showing a method of capturing an original image of a target area using a camera array with a wide-angle setting applied.
Figure 4 is a conceptual diagram showing a method of capturing an original image corresponding to a preset area of interest among the target areas using a camera array with telephoto settings applied.
Figure 5 is a video service providing device for monitoring a distant scene according to an embodiment of the present application, creates a panoramic image by stitching the original image, matches the panoramic images according to different angle of view settings, creates a database, and receives information from the client. This is a conceptual diagram to explain the process of playing video based on provided user input.
Figure 6 is a diagram illustrating a viewport that changes according to a user input provided from a client based on a reference panoramic image.
Figure 7 is a diagram showing an exemplary comparison of screens displayed on a client when different viewing angle settings are applied to the target area according to the same viewing direction.
Figure 8 is a schematic configuration diagram of a video service providing device for monitoring a distant scene according to an embodiment of the present application.
Figure 9 is an operation flowchart of a method of providing a video service for monitoring a distant scene according to an embodiment of the present application.

Below, with reference to the attached drawings, embodiments of the present application will be described in detail so that those skilled in the art can easily implement them. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In order to clearly explain the present application in the drawings, parts that are not related to the description are omitted, and similar reference numerals are assigned to similar parts throughout the specification.

Throughout this specification, when a part is said to be “connected” to another part, this means not only “directly connected” but also “electrically connected” or “indirectly connected” with another element in between. "Includes cases where it is.

Throughout this specification, when a member is said to be located “on”, “above”, “at the top”, “below”, “at the bottom”, or “at the bottom” of another member, this means that a member is located on another member. This includes not only cases where they are in contact, but also cases where another member exists between two members.

Throughout the specification of the present application, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components unless specifically stated to the contrary.

This application relates to an apparatus and method for providing video services for monitoring distant scenes. For example, this application is about an image shooting, synthesis, and service method for displaying images captured using camera arrays of various angles of view at different viewpoints for the same distant scene to suit the user's needs.

1 is a schematic configuration diagram of a remote video service providing system including a video service providing device for monitoring a distant scene according to an embodiment of the present application.

Referring to FIG. 1, the remote video service providing system 10 is a video service providing device 100 for monitoring a distant scene according to an embodiment of the present invention (hereinafter referred to as the 'service providing device 100'). ), a camera module 200, a database 300, and a user terminal 400.

The service providing device 100, the camera module 200, the database 300, and the user terminal 400 may communicate with each other through the network 20. The network 20 refers to a connection structure that allows information exchange between nodes such as terminals and servers. Examples of such networks 20 include the 3rd Generation Partnership Project (3GPP) network, Long Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), wifi network, Bluetooth network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc., but are not limited thereto.

The user terminal 400 includes, for example, a smartphone, a SmartPad, a tablet PC, etc., as well as a Personal Communication System (PCS), a Global System for Mobile communication (GSM), a Personal Digital Cellular (PDC), and a PHS ( Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) terminal It can be any type of wireless communication device, such as:

In the description of the embodiment of the present application, the user terminal 400 displays the viewing angle selected by the user of the user terminal 400, target area (distant area subject to monitoring, etc.), gaze direction, viewing point of view, etc. It may be a device for transmitting a user input including to the service providing device 100 and, in response, playing the image of the viewport transmitted by the service providing device 100 through a built-in (mounted) display and providing it to the user. there is.

Meanwhile, in the description of the embodiments of the present application, the user terminal 400 may be referred to differently as a client, and the service providing device 100 provides a long-distance video service initiated by the present application in conjunction with the client. It can be understood as a video server prepared for this purpose.

More specifically, referring to FIG. 1, the user terminal 400 displays an eyepiece (not shown) and a predetermined image at a position opposite the eyepiece (not shown) so that the user can view the image (image) of the object of observation with his or her eyes. It may include a telescope type device 400a including a display unit (not shown) provided to reproduce (images). Additionally, referring to FIG. 1, the user terminal 400 may broadly include a portable terminal type device 400b, a personal computer type device 400c, a Head Mount Display (HMD) type device, etc. according to an implementation example of the present application. there is.

In this regard, the way in which the user input transmitted from the user terminal 400 to the service providing device 100 is applied to the user terminal 400 may be determined differently depending on the type of the user terminal 400. Illustratively, the user input is a user selection input linked to specific viewing settings of the image (panoramic image, etc.) provided to the client by the long-distance video service providing system 10 disclosed herein, and as described later, viewing location information, It may include viewing direction information, shooting point of view information, angle of view setting information, etc., and the input method of each information (data) forming such user input may be determined differently depending on the type of user terminal 400.

For example, in the case of the telescope type device 400a, viewing direction information is applied according to the way the user physically rotates the body structure including the eyepiece (not shown) and the display unit (not shown), whereas in the case of the portable terminal In the case of the personal computer type device 400b, the viewing direction information may be applied in response to a touch input applied through the touch display of the user terminal 400, and in the case of the personal computer type device 400c, the viewing direction information may be applied through the personal computer type device 400c. ) may be applied in response to a manipulation input to a separate input device (eg, mouse, keyboard, etc.) provided in the device.

In addition, in the description of the embodiment of the present application, the camera module 200 may be installed with respect to the target area to capture an original image of the target area, and according to the implementation of the present application, the camera module 200 may be installed with respect to the target area to capture an original image of the target area. In order to obtain the original image, a plurality of camera modules 200 are arranged in relation to the target area to each photograph a plurality of divided areas according to a predetermined viewing angle setting, or as the number of installed camera modules 200 increases, Considering the cost, etc., a single camera module 200 is arranged for the target area, but a method of obtaining the original image by sequentially photographing a plurality of divided areas where the target area is divided according to a predetermined angle of view setting can also be applied. .

In this regard, when a plurality of camera modules 200 are arranged with respect to the target area, the short-definition camera module 200 sequentially photographs the divided areas demarcating the target area to obtain the original image individually. There can be an advantage in that real-time images of the target area can be quickly provided to the client without distortion because the entire image can be captured simultaneously.

In addition, in the description of the embodiment of the present application, the database 300 mutually stores panoramic images obtained by stitching original images according to each view angle setting obtained by the camera module 200 and panoramic images according to different view angle settings. It may be storage for classifying and storing the matched processed images according to the shooting time of each image. In this regard, the service providing device 100 processes the original image of the target image received from the camera module 200 and stores it in advance in the database 300, and allows the user to monitor the target area from the user terminal 400. When a user input associated with the viewing settings for is applied, an image matching the user input can be searched from the database 300 to extract a viewport, and the extracted viewport can be output through the user terminal 400. .

Hereinafter, specific functions and operations of the service providing device 100 will be described with reference to FIGS. 2 to 7.

The service providing device 100 may acquire a plurality of original images captured at a predetermined viewpoint corresponding to a plurality of different viewing angle settings for the target area. For reference, in the description of the embodiment of the present application, the target area is an area that is subject to filming an image provided by the service providing device 100 to the user terminal 400, which is a client, and is an area that the user wants to monitor through the image. It can mean a distant space. Here, that the target area is remote simply means that the target area is located relatively remotely based on the user's viewing position reflected in the image provided to the user terminal 400 to monitor the target area, and the user terminal 400 The actual physical distance between the user and the target area is not limited to being interpreted as a long distance.

To be more specific, the target area may include places that are inaccessible to the general public, such as border areas, mountainous areas, migratory bird habitats, etc. In this regard, in target areas such as the above-mentioned example, a specific local area of interest or object is observed from a distance using a telescope, etc., because in this case, the target area is observed at a narrow angle of view using a telescope from a distance. , it is not easy for viewers to search for areas or objects of particular interest among a wide range of distant areas using only telescope images. In addition, in cases where visibility is poor or at night due to the effects of fog, fine dust, rainfall, or snowfall, the quality of observation images using a telescope deteriorates, making it more difficult to search for and observe areas or objects of interest. do.

This problem can be solved in the following two aspects. First, if the imaging equipment through which the user observes a wide area at a distance can support images with multiple angles of view, images with a wide angle of view (for example, as described later) Confirm the location of the area of interest in an image captured through a wide field-of-view camera array, as shown in Figure 3, and enlarge the area of interest with a relatively narrow angle of view (e.g., as shown in Figure 2, which will be described later). Detailed observations can be made through images captured through a standard field-of-view camera array or images captured through a narrow field-of-view camera array as shown in FIG. 4, which will be described later. Additionally, in cases where visibility is poor, an image containing an area of interest that the user wishes to observe can be extracted and provided from an image captured and stored in advance.

In this regard, in order to provide the user with images according to various viewing angle settings for the target area, the service providing device 100 disclosed herein may determine a plurality of viewing angle settings to be provided in the observation service for the target area.

Specifically, according to an embodiment of the present application, the service providing device 100 has a standard angle of view setting (standard FoV) preset for the target area, a wide angle setting ( Three angle of view settings, including a wide FoV) and a telephoto setting (narrow FoV) to capture a narrow area compared to the area captured by the standard angle of view setting, can be determined as the type of angle of view to be provided by the observation service for the target area. It is not limited to this.

As another example, the service providing device 100 records the average distance from a predetermined reference viewing position to the target area, the size (area, volume, etc.) of the target area, the type of interest area existing within the target area, the number of interest areas, etc. Of course, it is possible to support two angles of view settings or support four or more angles of view settings according to the implementation example of the present application.

In relation to this, Figure 2 is a conceptual diagram showing a method of capturing the original image of the target area using a camera array with a standard angle of view setting, and Figure 3 shows an original image of the target area using a camera array with a wide angle setting applied. This is a conceptual diagram showing a shooting method, and Figure 4 is a conceptual diagram showing a method of shooting an original image corresponding to a preset area of interest among the target areas using a camera array with telephoto settings applied.

Referring to FIGS. 2 to 4, the service providing device 100 photographs a plurality of division areas in which the target area is divided according to each angle of view setting (e.g., standard angle of view setting, wide angle setting, telephoto setting, etc.). A plurality of original images may be received from a plurality of camera modules 200, but the present invention is not limited to this. As another example, the service providing device 100 may receive a plurality of original images sequentially photographed by a camera module 200 in a plurality of division areas in which the target area is divided according to each angle of view setting.

That is, although not shown in FIGS. 2 to 4, in cases where it is difficult to use a camera array, one camera module 200 according to each view angle setting is placed for the target area, but the target area, which is a large area at a distance, is used. You can capture the original video by sequentially scanning without missing anything.

Meanwhile, referring to Figures 2 to 4, the partial area divided for the same target area becomes larger in size (area) as the angle of view according to the angle of view setting becomes wider, and accordingly, the size (area) of the partial area divided by the same target area increases from the target area according to each angle of view setting. The number of segmented areas may increase as the angle of view becomes wider. Accordingly, as the angle of view is set to a relatively narrow angle of view, the number of camera modules 200 required to cover the entire target area may increase.

For example, the number of camera modules (S ₁ to S _n ) of the camera array required to photograph the same target area according to the standard view angle setting shown in FIG. It is a large value compared to the number of camera modules (W ₁ to W _n ) of the camera array required for filming, and the number of camera modules (N ₁ to N ) of the camera array required to film the same target area according to the telephoto setting shown in FIG. 4 It may be a small value compared to the number of _n ).

In other words, the position and pose of each camera module 200 included in the camera array can be set so that the target area, which is a wide area in the distance, can be captured without omission. As the angle of view of the set camera module 200 narrows, the distance becomes more visible. The size of the camera array to capture all of the target area, which is a large area, increases, and the cost accordingly also increases. Accordingly, referring to FIG. 4, when the angle of view for photographing the target area is narrowed to a certain level, it is possible to selectively photograph a wide area in the distance without exception or locally capture only images of a specific region of interest within the target region. During this process, all cameras shoot videos or images in the same way.

In this regard, according to an embodiment of the present application, the service providing device 100 determines that the area of the partial area partitioned for the target area in response to a predetermined viewing angle setting is less than or equal to a preset area or the number of partitioned partial areas is set to a preset threshold. If the value is greater than or equal to the value, the operation may be performed to partially obtain at least one original image corresponding to a preset area of interest among the target area, instead of photographing the entire target area.

Meanwhile, according to an embodiment of the present application, the area of interest is an area where major objects that the user is likely to want to observe appear in the target area (e.g., a specific tourist attraction, structure, etc. located within a specific area, a migratory bird sanctuary, etc. It can be set as an area where migratory birds mainly frequent, etc.).

In this regard, according to an embodiment of the present application, the service providing device 100 uses images acquired to capture the target area as a whole (for example, a plurality of original images acquired according to the above-described wide angle setting or standard viewing angle setting). It is possible to have an artificial intelligence-based image analysis model to detect certain key objects from (stitched panoramic images, etc.). For example, an artificial intelligence-based image analysis model may be a semantic segmentation model that is learned to identify the object type (class) corresponding to each pixel in the image, but is not limited to this. Rather, various artificial intelligence-based image analysis algorithm models that are already known or developed in the future can be applied here.

In addition, illustratively, main objects that can be identified in the target area may be set in various ways depending on the target area, such as types (classes) such as buildings, structures, specific terrain, animals and plants, and people, and these object types are as described in the examples above. It is not limited to

Meanwhile, the service providing device 100 sets the area of interest in which the main object appears as the area containing the area in which the main object is identified through statistical analysis based on an image analysis model for a plurality of image data acquired for the target area. You can.

Figure 5 is a video service providing device for monitoring a distant scene according to an embodiment of the present application, creates a panoramic image by stitching the original image, matches the panoramic images according to different angle of view settings, creates a database, and receives information from the client. This is a conceptual diagram to explain the process of playing video based on provided user input.

Referring to FIG. 5, the service providing device 100 displays a predetermined viewpoint (e.g., T1) in response to each view angle setting (e.g., standard view angle setting, wide angle setting, telephoto setting, etc.), as described in detail above. After obtaining a plurality of original images each taken at each point (view point, T2 viewpoint, T3 viewpoint, etc.), a plurality of original images obtained for each of the plurality of view angle settings can be stitched to generate a plurality of panoramic images.

Here, stitching refers to a technique of creating a single image (e.g., panoramic image, high-resolution image, etc.) by merging consecutive images (multiple original images) centered on the overlapping part, and this image stitching Since the technique is self-evident to those skilled in the art of image processing, detailed description will be omitted.

In other words, the service providing device 100 can generate a panoramic image captured at a specific point in time for each view angle setting through stitching of a plurality of original images. For example, if the reference time point at which the original image was acquired is divided into T1 time point to T3 time point, and the angle of view setting is classified into a standard angle of view setting, a wide angle setting, and a telephoto setting, the service providing device 100 sets the standard angle of view setting. Generate three panoramic images for each of the corresponding T1 to T3 viewpoints, generate three panoramic images for each of the T1 to T3 viewpoints corresponding to the wide-angle setting, and generate three panoramic images for each of the T1 to T3 viewpoints corresponding to the telephoto setting. Each of the three panoramic images can be created.

Additionally, the service providing device 100 may perform camera calibration to estimate the camera's intrinsic and extrinsic coefficients for each camera module 200 in the camera array for each view angle setting. Additionally, the service providing device 100 may generate a panoramic image by stitching a plurality of original images captured using a camera array for each view angle setting based on a preset reference image plane. For example, the reference image plane according to each angle of view setting may be set (designated) as the image plane corresponding to any one camera module 200 among the camera array that captures the original image corresponding to the angle of view setting.

In addition, referring to FIG. 5, the service providing device 100 converts an image stitched from a plurality of original images obtained in response to a relatively wide angle of view (for example, a wide angle setting) among a plurality of view angle settings into a reference panoramic image. You can set it.

Additionally, the service providing device 100 may register a plurality of panoramic images based on a set reference panoramic image. For example, when the angle of view setting is classified into a standard angle of view setting, a wide angle setting, and a telephoto setting, the service providing device 100 bases the panoramic image on a panoramic image generated from a plurality of original images acquired according to the standard angle of view setting. An operation of matching a panoramic image generated from a plurality of original images obtained according to a telephoto setting and an operation of matching a panoramic image based on a reference panoramic image can be performed.

That is, the service providing device 100 sets a panoramic image stitched from a camera array with the widest angle of view as a reference panoramic image for providing a remote monitoring service, and sets a panorama image stitched from a camera array with a relatively narrow angle of view compared to the reference panoramic image. The image can be registered with the reference panoramic image.

Meanwhile, registration information between the stitched panoramic image of each angle of view captured at a specific time point generated by the service providing device 100 and the reference image plane will be stored in the database 300 as shown in FIG. 5. You can. That is, the service providing device 100 stores registration information between a panoramic image captured and stitched at a plurality of different times using a camera array corresponding to each view angle setting and the reference image plane in the database 300, respectively. You can. Exemplarily, referring to FIG. 5, the database 300 includes a first database 301 for storing panoramic images taken according to a wide angle setting, stitched and matched, for each viewpoint, and a first database 301 for storing panoramic images taken according to a standard angle of view setting and stitched and matched. It may include a second database 302 for storing panoramic images for each viewpoint, and a third database 303 for storing panoramic images captured, stitched, and matched according to telephoto settings for each viewpoint.

Additionally, the service providing device 100 may receive a user input associated with viewing settings for monitoring the target area from the user terminal 400, which is a client. As described in detail above, a client connected to the service providing device 100 through the network 20 may be in the form of an electronic device terminal such as a laptop, tablet, or smartphone, a telescope for sightseeing, or a head mount (HMD). Display) may be provided in various forms, etc., depending on the embodiment of the present application. Meanwhile, the display built into the network client can reproduce the viewport transmitted by the service providing device 100.

Specifically, user input associated with viewing settings for monitoring the target area may include the user's viewing position information, viewing direction information, shooting viewpoint information, and angle of view setting information for the target area.

More specifically, the service providing device 100 provides information about the shooting time of the image that can be provided to the client (e.g., real-time (Live), T1 time, T2 time, T3 time, etc.) and user-selectable view angle settings ( can provide information about zoom level). In response, the user of the user terminal 400 selects the desired viewpoint and angle of view settings from the shooting viewpoint and angle of view settings provided by the service providing device 100, and provides viewing location information and viewing location information that changes depending on the movement of the client by the user. The client's pose information reflecting the viewing direction information may be transmitted to the service providing device 100. Here, the pose information may include yaw information and pitch information according to changes in the client's location or posture.

Additionally, the service providing device 100 may output a viewport through the user terminal 400 based on a panoramic image corresponding to a user input received from the user terminal 400 among the plurality of matched panoramic images.

Specifically, the service providing device 100 views a panoramic image captured and generated at a predetermined viewpoint corresponding to the capturing viewpoint information according to the viewpoint setting corresponding to the viewpoint setting information included in the user input among the plurality of matched panoramic images. The viewport corresponding to the location information and viewing direction information can be partially extracted and displayed.

In relation to this, Figure 6 is a diagram illustrating a viewport that changes according to a user input provided from a client based on a reference panoramic image.

Referring to FIG. 6, the service providing device 100 selects a panoramic image at the time of shooting received from the user terminal 400, and determines the user's gaze in the panoramic image from the client's pose information received from the user terminal 400. The location (real or virtual viewing location) can be calculated.

Meanwhile, according to an embodiment of the present application, when the network client starts providing an image service to the user, the center of the reference panoramic image may be placed at the center of the display of the user terminal 400. At this time, when the client's pose information (p ₁ ) at the service start time is called (yaw0, pitch0), and the client's pose information (p ₂ ) at a certain point in time is called (yaw1, pitch1), the viewport at the start time The center coordinates for (V ₁ ) and the center coordinates for the viewport (V ₂ ) at an arbitrary viewpoint can be calculated using Equations 1 and 2 below, respectively.

[Equation 1]

Viewport position at start point = (W/2, H/2)

[Equation 2]

Viewport position at a random point in time = (W/2+(yaw1-yaw0)*W/θ _h , H/2+(pitch1-pitch0)*H/θ _v )

Here, W is the number of pixels in the horizontal direction in the standard panoramic image, H is the number of pixels in the vertical direction in the standard panoramic image, θ _h is the vertical angle of view, and θ _v is the angle of view in the horizontal direction.

That is, the service providing device 100 displays the viewport position calculated from the pose information of the network client received from the user terminal 400 and the panoramic image generated according to the view angle setting (zoom level, FoV) selected by the user. The viewport position can be calculated, and a viewport can be created centered on the viewport position calculated from the panoramic image of the corresponding zoom level at the shooting point selected by the user and transmitted to the user terminal 400.

Meanwhile, when the user of the user terminal 400 selects the real-time shooting time setting, the service providing device 100 creates a viewport from the panoramic image stitched with the original image obtained from each camera array and displays it to the client. Can be transmitted.

In addition, according to an embodiment of the present application, a panoramic image generated according to the angle of view setting corresponding to the angle of view setting information included in the user input is generated based on at least one original image corresponding to a preset area of interest among the target areas. If it is an image (e.g., an image generated from an original image captured locally in an area of interest according to a telephoto setting as shown in FIG. 4), the service providing device 100 provides the viewing location information and the viewing location information included in the user input. A viewport for an area of interest that is relatively close to the viewing direction information may be displayed through the user terminal 400.

Figure 7 is a diagram showing an exemplary comparison of screens displayed on a client when different viewing angle settings are applied to the target area according to the same viewing direction. Specifically, Figure 7 (a) is a screen when the wide angle is set, Figure 7 (b) is a screen when the standard view angle is set, and Figure 7 (c) is a screen for the target area when the telephoto setting is set as an example. indicates

Figure 8 is a schematic configuration diagram of a video service providing device for monitoring a distant scene according to an embodiment of the present application.

Referring to FIG. 8 , the service providing device 100 may include an image collection unit 110, a stitching unit 120, a matching unit 130, and an image reproduction unit 140.

The image collection unit 110 may acquire a plurality of original images captured at a predetermined viewpoint corresponding to a plurality of different view angle settings for the target area.

According to an embodiment of the present application, the image collection unit 110 may receive a plurality of original images from a plurality of camera modules 200 that each photograph a plurality of division areas in which the target area is divided according to the angle of view setting. .

In addition, according to another embodiment of the present application, the image collection unit 110 may receive a plurality of original images sequentially photographed by a predetermined camera module 200 in a plurality of division areas in which the target area is divided according to the angle of view setting. You can.

As another example, the image collection unit 110 may acquire at least one original image corresponding to a preset area of interest among the target areas from the camera module 200.

The stitching unit 120 may generate a plurality of panoramic images by stitching the plurality of original images obtained for each of the plurality of view angle settings.

The matching unit 130 may set one panoramic image among the plurality of panoramic images as a reference panoramic image. Additionally, the matching unit 130 may match a plurality of panoramic images with each other based on a set reference panoramic image.

The video playback unit 140 may receive user input associated with viewing settings for monitoring the target area through the user terminal 400. Additionally, the image playback unit 140 may output a viewport based on a panoramic image corresponding to a received user input among the plurality of registered panoramic images.

Specifically, the image playback unit 140 captures and generates a video from among the plurality of matched panoramic images at a predetermined time point corresponding to the shooting point information included in the user input according to the view angle setting corresponding to the view angle setting information included in the user input. The viewport corresponding to the viewing location information and viewing direction information included in the user input can be partially extracted from the panoramic image and displayed.

In addition, according to an embodiment of the present application, a panoramic image generated according to the angle of view setting corresponding to the angle of view setting information included in the user input is generated based on at least one original image corresponding to a preset area of interest among the target areas. If it is an image, the image playback unit 140 may display, through the user terminal 400, a viewport for the area of interest that is relatively close to the viewing location information and viewing direction information included in the user input.

Below, we will briefly look at the operation flow of the present application based on the details described above.

Figure 9 is an operation flowchart of a method of providing a video service for monitoring a distant scene according to an embodiment of the present application.

The video service providing method for monitoring a distant scene shown in FIG. 9 may be performed by the service providing device 100 described above. Therefore, even if the content is omitted below, the content described with respect to the service providing device 100 can be equally applied to the description of the video service providing method for monitoring a distant scene.

Referring to FIG. 9 , in step S11, the image collection unit 110 may acquire (a) a plurality of original images captured in the target area at a predetermined point in time corresponding to a plurality of different view angle settings.

According to an embodiment of the present application, in step S11, the image collection unit 110 receives a plurality of original images from a plurality of camera modules 200 that each photograph a plurality of division areas in which the target area is divided according to the angle of view setting. can do.

In addition, according to another embodiment of the present application, in step S11, the image collection unit 110 generates a plurality of original images sequentially captured by a predetermined camera module 200 in a plurality of division areas in which the target area is divided according to the angle of view setting. can receive.

As another example, in step S11, the image collection unit 110 may acquire at least one original image corresponding to a preset area of interest among the target areas from the camera module 200.

Next, in step S12, the stitching unit 120 may (b) generate a plurality of panoramic images by stitching the plurality of original images obtained for each of the plurality of view angle settings.

Next, in step S13, the matching unit 130 (c) sets one of the plurality of panoramic images as a reference panoramic image, and can mutually match the plurality of panoramic images based on the set reference panoramic image. there is.

Next, in step S14, the video playback unit 140 may (d) receive a user input associated with viewing settings for monitoring the target area.

Next, in step S15, the image playback unit 140 may (e) output a viewport based on the panoramic image corresponding to the user input received in step S14 among the plurality of registered panoramic images.

Specifically, in step S15, the image playback unit 140 selects a predetermined time point corresponding to the shooting point information included in the user input according to the view angle setting corresponding to the view angle setting information included in the user input among the plurality of registered panoramic images. A viewport corresponding to the viewing location information and viewing direction information included in the user input can be partially extracted from the captured and generated panoramic image and displayed.

In addition, according to an embodiment of the present application, a panoramic image generated according to the angle of view setting corresponding to the angle of view setting information included in the user input is generated based on at least one original image corresponding to a preset area of interest among the target areas. If it is an image, in step S15, the image player 140 may display, through the user terminal 400, a viewport for the region of interest that is relatively close to the viewing location information and viewing direction information included in the user input.

In the above description, steps S11 to S15 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

A method of providing an image service for monitoring a distant scene according to an embodiment of the present application may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. . The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and constructed for the present invention or may be known and usable by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the method of providing a video service for monitoring a distant scene according to an embodiment of the present application described above may also be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The description of the present application described above is for illustrative purposes, and those skilled in the art will understand that the present application can be easily modified into other specific forms without changing its technical idea or essential features. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

10: Long-distance video service provision system
100: Video service providing device for monitoring distant scenes
110: Video collection unit
120: Stitching part
130: matching part
140: Video playback unit
200: Camera module
300: database
400: user terminal, client
20: Network

Claims (16)

In a method of providing video services for monitoring distant scenes,
(a) acquiring a plurality of original images captured at a predetermined viewpoint corresponding to a plurality of different viewing angle settings of the target area;
(b) generating a plurality of panoramic images by stitching the plurality of original images obtained for each of the plurality of view angle settings;
(c) setting one of the plurality of panoramic images as a reference panoramic image and matching the plurality of panoramic images based on the reference panoramic image;
(d) receiving user input associated with viewing settings for monitoring the target area; and
(e) outputting a viewport based on a panoramic image corresponding to the user input among the plurality of matched panoramic images,
Including,
In step (a),
If the area of the partial area delimited with respect to the target area in response to the predetermined viewing angle setting is less than or equal to a preset area or the number of partial areas is greater than or equal to a preset threshold value, at least one of the target areas corresponds to a preset area of interest. Obtaining the original image of
In step (d), the user input includes pose information including yaw information and pitch information according to changes in the user's position and posture reflecting the user's viewing position information and viewing direction information for the target area, and a shooting point. Contains information and view angle setting information,
In step (e),
Among the plurality of matched panoramic images, a panoramic image generated by being photographed at a predetermined viewpoint corresponding to the capturing viewpoint information according to an angle of view setting corresponding to the angle of view setting information, and the viewing position information within the panoramic image from the pose information, and Calculate the viewing direction information, partially extract and display the viewport corresponding to the viewing position information and the viewing direction information,
If the panoramic image generated according to the angle of view setting corresponding to the angle of view setting information is an image generated based on at least one original image corresponding to a preset area of interest among the target areas,
In step (e),
A method of providing a service, wherein the viewport for the area of interest that is relatively close to the viewing location information and the viewing direction information is displayed. According to paragraph 1,
In step (c), the reference panoramic image is,
A method of providing a service, characterized in that the image is stitched from the plurality of original images obtained corresponding to a relatively wide angle of view among the plurality of view angle settings. According to paragraph 1,
In step (a),
A method of providing a service, wherein the plurality of original images are respectively received from a plurality of camera modules that each photograph a plurality of division areas in which the target area is divided according to the angle of view setting. According to paragraph 1,
In step (a),
A method of providing a service, wherein a plurality of original images are sequentially photographed by a predetermined camera module in a plurality of division areas in which the target area is divided according to the angle of view setting. delete According to paragraph 1,
The plurality of view angle settings are:
A standard angle of view setting that is preset for the target area, a wide angle setting for shooting a wider area compared to the area captured by the standard angle of view setting, and a telephoto setting for shooting a narrow area compared to the area captured by the standard angle of view setting. Including methods of providing services. delete delete In a video service providing device for monitoring distant scenes,
An image collection unit that acquires a plurality of original images captured at a predetermined viewpoint corresponding to a plurality of different view angle settings of the target area;
a stitching unit for generating a plurality of panoramic images by stitching the plurality of original images obtained for each of the plurality of view angle settings;
a matching unit that sets one of the plurality of panoramic images as a reference panoramic image and matches the plurality of panoramic images based on the reference panoramic image; and
An image playback unit that receives user input associated with viewing settings for monitoring the target area and outputs a viewport based on a panoramic image corresponding to the user input among the plurality of matched panoramic images;
Including,
The video collection unit,
If the area of the partial area delimited with respect to the target area in response to the predetermined viewing angle setting is less than or equal to a preset area or the number of partial areas is greater than or equal to a preset threshold value, at least one of the target areas corresponds to a preset area of interest. Obtaining the original image of
The user input includes pose information including yaw information and pitch information according to changes in the user's position and posture reflecting the user's viewing position information and viewing direction information for the target area, shooting viewpoint information, and angle of view setting information. Contains,
The video playback unit,
Among the plurality of matched panoramic images, a panoramic image generated by being photographed at a predetermined viewpoint corresponding to the capturing viewpoint information according to an angle of view setting corresponding to the angle of view setting information, and the viewing position information within the panoramic image from the pose information, and Calculate the viewing direction information, partially extract and display the viewport corresponding to the viewing position information and the viewing direction information,
If the panoramic image generated according to the angle of view setting corresponding to the angle of view setting information is an image generated based on at least one original image corresponding to a preset area of interest among the target areas,
The video playback unit,
A service providing device that displays the viewport for the region of interest that is relatively close to the viewing location information and the viewing direction information. According to clause 9,
The service providing device is characterized in that the reference panoramic image is an image stitched from the plurality of original images obtained corresponding to a relatively wide angle of view among the plurality of view angle settings. According to clause 9,
The video collection unit,
Each of the plurality of original images is received from a plurality of camera modules that each capture a plurality of division areas in which the target area is divided according to the angle of view setting, or a plurality of division regions in which the target area is divided according to the angle of view setting are predetermined. A service providing device that receives the plurality of original images sequentially captured by a camera module. delete According to clause 9,
The plurality of view angle settings are:
A standard angle of view setting that is preset for the target area, a wide angle setting for shooting a wider area compared to the area captured by the standard angle of view setting, and a telephoto setting for shooting a narrow area compared to the area captured by the standard angle of view setting. Including, a service provision device. delete delete A computer-readable recording medium recording a program for executing the method according to any one of claims 1 to 4 and 6 on a computer.

Images