KR102278748B1 - User interface and method for 360 VR interactive relay - Google Patents

Abstract

A user interface and method for 360 VR interactive relay are provided. A user interface method for 360 VR interactive relay according to an embodiment of the present invention comprises: receiving a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and sub-images for a plurality of videos; generating a screen image in which the user's region of interest consists of an ultra-high-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, and a sub-image is added to the peripheral region; and outputting the generated screen image.
As a result, 360 VR relay and multiple images to which tile-based streaming service technology is applied can be played on one screen, and when viewing a 360 VR video, a specific sub media is selected from a plurality of added sub media to switch the playing screen By doing so, it is possible to provide convenience to the user.

Description

User interface and method for 360 VR interactive relay

The present invention relates to a user interface and method, and more particularly, to a user interface and method for 360 VR interactive relay.

Virtual or augmented reality technologies that wear HMDs are developing very rapidly. Such virtual reality is a technology that allows a user to enter a virtual space, and representative devices to which it is applied include Oculus Rift and Vive, and a dedicated controller is used as an input device for content manipulation.

Watching a 360VR image and a panoramic image by wearing such an HMD does not view the entire area at once, but only a partial area (Field of View; FOV) at a time.

That is, in order to view the entire area, a partial area is viewed while moving up, down, left, and right little by little. The user's voice recognition, view, etc. can be recognized to control the display screen.

On the other hand, in order to provide a streaming service of 8K (7680x4320)/16K (15360x8640) class 360VR and super-capacity video such as panoramic video, rather than streaming super-capacity video data all at once, the video is divided into split images (tiles) in advance, A tile-based streaming service technology, which is a method of adaptively streaming only the viewing area, has been developed, but a user interface for selectively viewing multiple videos or playing them in one screen is not provided, which is inconvenient. .

Therefore, when viewing 360 VR video to which tile-based streaming service technology is applied, it is required to find a way to selectively view multiple videos or to play them in one screen.

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a 360 VR relay to which tile-based streaming service technology is applied and a user interface for 360 VR interactive relay that can play multiple images in one screen. and to provide a method.

Another object of the present invention is to provide a user interface and method for 360 VR interactive relay that selects a specific sub media from among a plurality of added sub media and switches a screen being played when watching a 360 VR video.

According to an embodiment of the present invention for achieving the above object, a user interface method for 360 VR interactive relay is a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and a sub-image for a plurality of videos is received. step; generating a screen image in which the user's region of interest consists of an ultra-high-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, and a sub-image is added to the peripheral region; and outputting the generated screen image.

And according to an embodiment of the present invention, a user interface method for 360 VR interactive relay includes: setting a user's ROI according to the user's view, and setting a region other than the set ROI as a peripheral region; may further include.

In addition, the sub image may include advertisement information having a commercial purpose or information on a recommended image related to the image.

In addition, the video is a tile-based MPEG image, and the sub-image may have a resolution of 1920x1080 when the resolution of the split image or low-resolution image for each tile of the ultra-high-resolution or high-resolution image constituting the screen image is 1024X1024.

In addition, the outputting of the screen image includes, when the user's ROI is set, a region within a preset range from the set ROI is set as the first peripheral region, and a region other than the ROI and the first peripheral region is set as the second peripheral region. When the region of interest is set as a region and the region of interest is configured as an ultra-high-resolution image, a screen image in which the first peripheral region is configured as a high-resolution image and the second peripheral region is configured as a low-resolution image may be output.

And, in the user interface method for 360 VR interactive relay according to an embodiment of the present invention, when the view of the screen moves or the user selects a specific object by moving the cursor while the screen image is output, The method may further include a step of updating the output screen image.

In addition, in the step of updating the screen image, the viewpoint of the screen may be moved or the cursor may be moved according to the movement of the wearable device worn on the body other than the user's head.

In the step of updating the screen image, when the specific object selected by the cursor is a sub-image in a state in which the screen image is output, an ultra-high-resolution image or a high-resolution image generated based on the same video as the selected sub-image is output to the region of interest and a low-resolution image generated based on the same video as the selected sub-image may be output to the peripheral area.

In addition, in the step of updating the screen image, when the specific object selected by the cursor is a sub-image while the screen image is output, an ultra-high-resolution image or a high-resolution image generated based on the same video as the selected sub-image is placed in the region of interest. A low-resolution image generated based on the video output and the same video as the selected sub-image is output to the peripheral region, but the sub-image generated based on the same video as the super-high-resolution image or high-resolution image that was output to the region of interest is selected. It can be added and outputted in the place of the outputted peripheral area.

On the other hand, according to another embodiment of the present invention, a user interface for 360 VR interactive relay includes: a communication unit for receiving a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and a sub-image for a plurality of videos; a processor for generating a screen image in which the user's region of interest is composed of a super-resolution or high-resolution image, and the peripheral region of the region of interest is composed of a low-resolution image, and a sub-image is added to the peripheral region; and a display for outputting a screen image.

On the other hand, a user interface method for 360 VR interactive relay according to another embodiment of the present invention comprises the steps of: generating a low-resolution image, an ultra-high-resolution image or a high-resolution image, and a sub-image for each video; and selecting a low-resolution image and an ultra-high-resolution image or a high-resolution image for a specific video, so that the user's region of interest consists of an ultra-high-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, but a sub image for another video and outputting a screen image added to the peripheral area.

On the other hand, according to another embodiment of the present invention, a user interface for 360 VR interactive relay includes a processor for generating a low-resolution image, an ultra-high-resolution image or a high-resolution image, and a sub-image for each video; and selecting a low-resolution image and an ultra-high-resolution image or a high-resolution image for a specific video, so that the user's region of interest consists of an ultra-high-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, but a sub image for another video and a display for outputting a screen image added to the surrounding area.

As described above, according to embodiments of the present invention, 360 VR relay and multiple images to which tile-based streaming service technology is applied can be played in one screen, and when viewing 360 VR images and audio, a plurality of sub-media Convenience can be provided to the user by selecting a specific sub media from among them and switching the screen being played.

1 is a diagram provided in the description of a user interface system for 360 VR interactive relay according to an embodiment of the present invention;
2 is a diagram illustrating a user interface for playing a 360 VR image;
3 is a diagram illustrating a user interface for controlling a 360 VR image being played;
4 is a diagram illustrating a user interface for controlling a 360 VR image being played through an HMD;
5 is a view illustrating a screen image in which a 360 VR video is being played, and
6 is a diagram illustrating a user interface for converting a 360 VR image being played back to sub media.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

1 is a diagram provided to explain a user interface system for 360 VR interactive relay according to an embodiment of the present invention. As shown in FIG. 1 , the user interface system for 360 VR interactive relay according to an embodiment of the present invention includes a processor 100 , a display 200 , a communication unit 300 and a storage unit 400 , and includes a tile 360 VR relay and multiple videos to which the base streaming service technology is applied can be played on one screen, and when viewing a 360 VR video, you can select a specific sub media from among a plurality of added sub media to switch the playing screen.

The processor 100 may generate a screen image in which the user's region of interest is composed of an ultra-high resolution or high-resolution image, and the peripheral region of the region of interest is composed of a low-resolution image, and a sub-image is added to the peripheral region.

For example, when the processor 100 receives data for a low-resolution image, an ultra-high-resolution image, a high-resolution image, and a sub-image from the server through the communication unit 300, the user's region of interest according to the user's view By setting a region other than the set region of interest as a peripheral region, the user's region of interest is composed of a super-resolution or high-resolution image, and a screen image in which a sub-image is added to the peripheral region of the region of interest can be generated.

In this case, the sub-image may include an image that relays a relay image output through a screen image from another view, advertisement information having a commercial purpose, or information on a recommended image related to the image.

In addition, the video is a tile-based MPEG image, and when the resolution of the split image or low-resolution image for each tile of the ultra-high-resolution or high-resolution image constituting the screen image is 1024X1024, the resolution may be 1920X1080.

For another example, the processor 100, through the communication unit 300, after receiving the data for the low-resolution image, the ultra-high-resolution image, the high-resolution image and the sub-image from the server, when the user's region of interest is set, the set interest When a region within a preset range from the region is set as the first peripheral region, and a region other than the region of interest and the first peripheral region is set as the second peripheral region, and the region of interest is composed of an ultra-high-resolution image, the first peripheral region is It is possible to generate a first image composed of a high-resolution image and a second peripheral region composed of a low-resolution image.

For another example, the processor 100 generates a low-resolution image, an ultra-high-resolution image or a high-resolution image, and a sub-image for each video, sets the user's ROI according to the user's view, and sets the set ROI. In addition, after setting the region as the peripheral region, the user’s region of interest is composed of a super-resolution or high-resolution image by selecting a low-resolution image, an ultra-high resolution image, or a high-resolution image for a specific video, and the peripheral region of the region of interest is converted to a low-resolution image. However, it is possible to create a screen image in which a sub-image for another video is added to the peripheral area.

In addition, the processor 100 generates a new screen image and updates the output screen image when a view point of the screen moves or a user selects a specific object by moving a cursor while the screen image is output. can make it happen

In this case, when the processor 100 is wirelessly connected to a wearable device that is separately provided and worn on a body other than the user's head through the communication unit 300, the screen is displayed according to the movement of the wearable device worn on the body other than the user's head. By moving the viewpoint or the cursor, it is possible to provide convenience to the user.

In addition, the processor 100 calculates the user's viewpoint in a state in which the screen image is output, and when a specific object selected by the cursor is a sub-image, an ultra-high-resolution image or high-resolution image generated based on the same video as the selected sub-image An image may be output to the region of interest, and a low-resolution image generated based on the same video as the selected sub-image may be output to the peripheral region.

In addition, the processor 100 may output an ultra-high-resolution image output to the region of interest or a sub-image generated based on the same video as the high-resolution image is added to the position of the peripheral region where the selected sub-image is output.

The display 200 is implemented as an HMD (Head Mounted Display) worn on the user's head or a display 200 capable of outputting 360 VR images, so as to reproduce 360 VR relay and multiple images in one screen. A screen image can be output.

The communication unit 300, in a module wirelessly connected to the server, receives data for a 360 VR image from the server, is wirelessly connected to a wearable device that is separately provided and worn on the body other than the user's head, and receives data. means, and the storage unit 400 is a storage medium that provides a storage space necessary for the processor 100 to operate.

In more detail, the communication unit 300 may receive a low-resolution image, an ultra-high-resolution image, or a high-resolution image for a specific video and a sub-image for a plurality of videos.

2 is a diagram illustrating a user interface for playing a 360 VR image.

The user interface system for 360 VR interactive relay according to an embodiment of the present invention may provide a user interface through the display 200 .

Specifically, the display 200 according to an embodiment of the present invention may provide a user interface for playing a 360 VR image as illustrated in FIG. 2 .

The user interface provided by the display 200 to reproduce the 360 VR image includes an A-1 area 211 where a server URL can be input, an A-2 area 212 where a server port number can be input, and a specific image. In order to select the playback start point (time) of the segment index, the segment index input A-3 area 213, TV output mode A-4 area 214, HMD output mode selectable It may include an A-5 area 215 and an A-6 area 216 in which the selected 360 VR image can be reproduced.

3 is a diagram illustrating a user interface for controlling a 360 VR image being reproduced, FIG. 4 is a diagram illustrating a user interface for controlling a 360 VR image being played through an HMD, and FIG. 5 is a 360 VR image It is a diagram exemplifying the screen image being reproduced.

The display 200 according to an embodiment of the present invention may provide a user interface for controlling the 360 VR image being reproduced as illustrated in FIGS. 3 to 5 .

In order to control the 360 VR image being reproduced, the user interface provided by the display 200 outputs a sub image for selecting and playing one or more sub media, so that a sub image to be converted into a screen image can be selected. The -1 area 221, the B-2 area 222 where the screen image constituting the 360 VR image is output, the B-3 area 223 where the menu group can be turned on/off, the sound volume can be controlled The B-4 area 224, the B-5 area 225 that allows you to select the FULL mode that reproduces the image of the same quality without setting the 360 VR image as the region of interest and the surrounding region, and the 360 VR image as the region of interest and B-6 area 226, screen where the user can select a VR mode that reproduces an image in which the user's area of interest is composed of a super-resolution or high-resolution image by setting the area as a peripheral area and setting the area as a peripheral area other than the set area of interest A B-7 region 227 capable of adjusting the number of sub-images added to the image may be included.

6 is a diagram illustrating a user interface for converting a 360 VR image being played back to sub media.

The display 200 according to an embodiment of the present invention may provide a user interface for converting a 360 VR image into sub media as illustrated in FIG. 6 .

Basically, depending on the user's point of view (region of interest), super-resolution or high-resolution images and low-resolution images are played around the ROI.

In order to convert the 360 VR image to sub media, the user interface provided by the display 200 provides a C-1 region 231 that can end the image being played, and a low-resolution image composed of a low-resolution image among 360 VR images is played. C-2 area 232 in the middle of being played, C-3 area 233 where you can select a sub image for selecting and playing sub media, and super high resolution or high resolution video composed of super high resolution or high resolution images among 360 VR images are being played. C-4 region 234 may be included.

That is, the user uses the user interface for converting the 360 VR image to sub media, and when the screen image is output and the specific object selected by the cursor is the sub image, the generated image is generated based on the same video as the selected sub image. An ultra-high-resolution image or a high-resolution image may be output to the region of interest, and a low-resolution image generated based on the same video as the selected sub-image may be output to the peripheral region.

In this case, an ultra-high-resolution image output to the region of interest or a sub-image generated based on the same video as the high-resolution image may be added to and outputted at the position of the peripheral region where the selected sub-image is output.

Through this, when viewing a 360 VR video, it is possible to select a specific sub media from among a plurality of added sub media to switch the playing screen.

On the other hand, it goes without saying that the technical idea of the present invention can also be applied to a computer-readable recording medium containing a computer program for performing the functions of the apparatus and method according to the present embodiment. In addition, the technical ideas according to various embodiments of the present invention may be implemented in the form of computer-readable codes recorded on a computer-readable recording medium. The computer-readable recording medium may be any data storage device readable by the computer and capable of storing data. For example, the computer-readable recording medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, hard disk drive, or the like. In addition, the computer-readable code or program stored in the computer-readable recording medium may be transmitted through a network connected between computers.

In addition, although preferred embodiments of the present invention have been illustrated and described above, the present invention is not limited to the specific embodiments described above, and the technical field to which the present invention belongs without departing from the gist of the present invention as claimed in the claims Various modifications are possible by those of ordinary skill in the art, and these modifications should not be individually understood from the technical spirit or prospect of the present invention.

100 : processor
200: display
300: communication department
400: storage

Claims (12)

Receiving a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and sub-images for a plurality of videos;
generating a screen image in which the region of interest consists of a super-resolution or high-resolution image by calculating the user's viewpoint, and the peripheral region of the region of interest consists of a low-resolution image, and a sub-image is added to the peripheral region; and
Including; outputting the generated screen image;
The step of outputting the screen image is,
When the user's ROI is set, a region within a preset range from the set ROI is set as the first peripheral region, and a region other than the ROI and the first peripheral region is set as the second peripheral region, so that the ROI becomes super high-resolution When composed of an image, the user interface method for 360 VR interactive relay, characterized in that the first peripheral region is composed of a high-resolution image, and the second peripheral region is composed of a low-resolution image to output a screen image.
The method according to claim 1,
The user interface method for 360 VR interactive relay further comprising; setting the user's ROI according to the user's viewpoint, and setting the region other than the set ROI as a peripheral region.
3. The method according to claim 2,
sub image,
A user interface method for 360 VR interactive relay, characterized in that it includes information on advertisement information having a commercial purpose or a recommended image related to the image.
3. The method according to claim 2,
the video,
It is a tile-based MPEG video,
sub image,
A user interface method for 360 VR interactive relay, characterized in that the resolution is 1920X1080 when the resolution of the split image or low-resolution image for each tile of the super-resolution or high-resolution image constituting the screen image is 1024X1024.
delete The method according to claim 1,
360 VR interactive relay, characterized in that the 360 VR interactive relay further comprises; when the view of the screen moves in a state in which the screen image is output or the user selects a specific object by moving the cursor, the output screen image is updated; User interface method for .
Receiving a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and sub-images for a plurality of videos;
generating a screen image in which the region of interest consists of a super-resolution or high-resolution image by calculating the user's viewpoint, and the peripheral region of the region of interest consists of a low-resolution image, and a sub-image is added to the peripheral region; and
Including; outputting the generated screen image;
In a state in which the screen image is output, the step of updating the output screen image when the view point of the screen moves or when the user selects a specific object by moving the cursor; further comprising,
The step of updating the screen image is,
A user interface method for 360 VR interactive relay, characterized in that the view point of the screen moves or the cursor moves according to the movement of the wearable device worn on the body other than the user's head.
Receiving a low-resolution image, an ultra-high-resolution image or a high-resolution image for a specific video, and sub-images for a plurality of videos;
generating a screen image in which the region of interest consists of a super-resolution or high-resolution image by calculating the user's viewpoint, and the peripheral region of the region of interest consists of a low-resolution image, and a sub-image is added to the peripheral region; and
Including; outputting the generated screen image;
In a state in which the screen image is output, the step of updating the output screen image when the view point of the screen moves or when the user selects a specific object by moving the cursor; further comprising,
The step of updating the screen image is,
When the screen image is output and the specific object selected by the cursor is a sub-image, an ultra-high-resolution image or a high-resolution image generated based on the same video as the selected sub-image is output to the region of interest, and the same video as the selected sub-image is displayed. A user interface method for 360 VR interactive relay, characterized in that the low-resolution image generated based on it is output to the surrounding area.
9. The method of claim 8,
The step of updating the screen image is,
When the screen image is output and the specific object selected by the cursor is a sub-image, an ultra-high-resolution image or a high-resolution image generated based on the same video as the selected sub-image is output to the region of interest, and the same video as the selected sub-image is displayed. Let the low-resolution image generated based on it be output in the surrounding area,
User interface for 360 VR interactive relay, characterized in that the sub-image generated based on the same video as the super-resolution image that was output to the region of interest or the high-resolution image is added to the position of the peripheral region where the selected sub-image was outputted Way.
a communication unit for receiving a low-resolution image, an ultra-high-resolution image, or a high-resolution image for a specific video and a sub-image for a plurality of videos;
a processor for generating a screen image in which the user's region of interest is composed of a super-resolution or high-resolution image, and the peripheral region of the region of interest is composed of a low-resolution image, and a sub-image is added to the peripheral region; and
Including; a display for outputting a screen image;
The processor is
When the user's ROI is set, a region within a preset range from the set ROI is set as the first peripheral region, and a region other than the ROI and the first peripheral region is set as the second peripheral region, so that the ROI becomes super high-resolution When composed of an image, the user interface for 360 VR interactive relay, characterized in that the first peripheral region is composed of a high-resolution image, and the second peripheral region is composed of a low-resolution image to output a screen image through the display.
generating a low-resolution image, an ultra-high-resolution image or a high-resolution image, and a sub-image for each video; and
By selecting a low-resolution image and an ultra-high-resolution image or a high-resolution image for a specific video, the user's region of interest consists of a super-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, but the sub-image for another video is outputting a screen image added to a peripheral area; including,
In a state in which the screen image is output, the step of updating the output screen image when the view point of the screen moves or when the user selects a specific object by moving the cursor; further comprising,
The step of updating the screen image is,
A user interface method for 360 VR interactive relay, characterized in that the view point of the screen moves or the cursor moves according to the movement of the wearable device worn on the body other than the user's head.
a processor for generating a low-resolution image, an ultra-high-resolution image or a high-resolution image, and a sub-image for each video; and
By selecting a low-resolution image and an ultra-high-resolution image or a high-resolution image for a specific video, the user's region of interest consists of a super-resolution or high-resolution image, and the peripheral region of the region of interest consists of a low-resolution image, but the sub-image for another video is Including; a display for outputting a screen image added to the peripheral area;
The processor is
When the view of the screen moves while the screen image is output or a specific object is selected by the user moving the cursor, the output screen image is updated,
The processor is
A user interface for 360 VR interactive relay, characterized in that when the screen image is updated, the viewpoint of the screen moves or the cursor moves according to the movement of the wearable device worn on the body other than the user's head.

Images