KR102261597B1 - Apparatus for processing caption of virtual reality video content - Google Patents

Abstract

The present invention relates to a caption processing device for VR video content, wherein the device for captioning VR video content according to the present invention includes an image reproducing unit ( 10); a subtitle layer generator 20 for generating a curved subtitle layer 21 in which a subtitle corresponding to the VR image content is displayed on a virtual curved surface curved with a predetermined curvature satisfying the following <Condition 1>; a subtitle reproducing unit 30 for reproducing the curved subtitle layer 21; a subtitle overlay unit 40 that overlays the reproduced curved subtitle layer 21 on the image layer 11; and a VR display unit 50 that simultaneously displays VR image content and subtitles on a VR screen.
<Condition 1>
Ki×80% ≤ Kc ≤ Ki×120%
Here, Ki is the curvature of the image layer 11 , and Kc is the curvature of the curved caption layer 21 .

Description

Caption processing device for VR video content {APPARATUS FOR PROCESSING CAPTION OF VIRTUAL REALITY VIDEO CONTENT}

The present invention relates to a device for processing captions for VR video content, and more particularly, to a device for processing captions used for VR video content by being applied to a VR device that reproduces virtual reality (VR) video content. .

Virtual reality (VR) is a technology that makes a specific environment or situation that is difficult to experience daily through computing technology virtual and supports interaction with the user, making it as if the user is actually experiencing the environment or situation. . Although such virtual reality was initially used for surgical practice or training on how to control tanks and aircraft, it has recently been expanded from entertainment industries such as games, movies, and theme parks to various industries such as education and healthcare. As VR devices that provide such virtual reality, Ogulus' Rift, HTC's Vive, Samsung's Gear VR, Google's Guard Board, etc. have been released, and related technical contents are also disclosed in the patent documents of the following prior art documents.

1 is a perspective view schematically showing a conventional virtual reality (VR) device. Referring to this, an example of a VR device will be described. The VR device 1 is composed of an HMD frame 2 and a screen insertion frame 3 However, in a state in which the VR device 1 is mounted on the user's head through the HMD frame 2 and the smart terminal 4 is inserted and fixed in the screen insertion frame 3 provided in the front part, the smart terminal 4 Users can watch virtual reality content by running the VR content playback app installed in the .

However, since general VR video content does not support subtitles, there is a problem in that it is difficult to acquire information about the content or to understand the content. On the other hand, some VR video content is used as a caption by writing a text image on the video, but in this case, since the text image is fixed at a predetermined position in the video, the caption becomes invisible when the user's field of view is moved. In addition, a flat layer can be placed in front of the VR video content layer and subtitles can be displayed on the flat layer, but in this case, the sharpness and readability of the subtitles deteriorates depending on the distance between the subtitles and the video content layer, and when watching for a long time, There is a problem of increasing eye fatigue.

Accordingly, there is an urgent need for a method to solve the problems of the prior art for supporting subtitles in VR video content.

US 10-1789153 B1

The present invention is to solve the above problems of the prior art, and one aspect of the present invention is to provide a subtitle processing device for VR image content that overlays a subtitle layer curved with the same curvature on a VR image layer reproduced on a spherical curved surface. is to provide

Another aspect of the present invention is a method that converts subtitle data input in the form of a flat subtitle layer into a curved subtitle layer, and extracts text merged into a text image in VR video content to create a curved subtitle layer based on this. It is intended to provide a subtitle processing device for VR video content.

Another aspect of the present invention is to provide a caption processing device for VR image content capable of detecting a moving user's field of view and moving a caption layer accordingly.

An apparatus for processing VR image content according to an embodiment of the present invention includes: an image reproducing unit that reproduces an image layer in which VR image content is displayed on a virtual spherical curved surface; a subtitle layer generator generating a curved subtitle layer in which a subtitle corresponding to the VR image content is displayed on a virtual curved surface curved with a predetermined curvature satisfying the following [Condition 1]; a subtitle reproducing unit for reproducing the curved subtitle layer; a subtitle overlay unit that overlays the reproduced curved subtitle layer on the image layer; and a VR display unit that simultaneously displays the VR video content and the subtitles on a VR screen.

<Condition 1>

Ki×80% ≤ Kc ≤ Ki×120%

Here, Ki is the curvature of the image layer, and Kc is the curvature of the curved caption layer.

In addition, in the apparatus for processing VR video content according to an embodiment of the present invention, the apparatus further comprises: a subtitle storage unit for storing a flat subtitle layer in which the subtitle is displayed on a virtual plane, wherein the subtitle layer unit comprises: A flat caption layer may be converted into the curved caption layer.

In addition, in an apparatus for processing VR video content according to an embodiment of the present invention, there is provided an apparatus comprising: an image storage unit for storing the VR video content in which a text image is incorporated; a text extraction unit for recognizing and extracting the text image from the stored VR video content; and a text data conversion unit converting the extracted text image into text data, wherein the caption layer unit may generate the curved caption layer based on the converted text data.

In addition, the apparatus for processing VR video content according to an embodiment of the present invention may further include a caption moving unit that moves the position of the reproduced curved caption layer based on the user's viewpoint.

In addition, in the device for processing VR video content according to an embodiment of the present invention, at least one of the size, position, color, font, kerning, length, margin, brightness, and line spacing of the subtitle displayed on the curved subtitle layer. It may further include; a subtitle option unit for converting any one or more.

Also, in the device for processing VR video content according to an embodiment of the present invention, the VR video content may be installed in a smart terminal in the form of a VR video content playback app and played by a user's selection.

Also, in the device for processing VR video content according to an embodiment of the present invention, the caption laber generating unit may be mounted on a smart terminal that plays the VR video content in the form of a VR video content playback app.

In addition, there is provided an apparatus for processing captions for VR video content according to an embodiment of the present invention, comprising: an area of interest identification unit for identifying a predetermined area of interest on the reproduced VR video content; and a subtitle search unit that, when the region of interest is identified, searches for the corresponding subtitle and transmits the retrieved subtitle to the subtitle layer generator.

Also, in the apparatus for processing VR video content for captions according to an embodiment of the present invention, the ROI identification unit may identify the ROI when the ROI exists in a user's viewing angle range.

In addition, in the device for processing VR video content according to an embodiment of the present invention, the apparatus further includes a pupil detection unit configured to detect the user's point of view by detecting the position of the user's pupil, wherein the region of interest identification unit comprises: The ROI may be identified based on viewpoint information of .

The features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

Prior to this, the terms or words used in the present specification and claims should not be construed in a conventional and dictionary meaning, and the inventor may properly define the concept of the term to describe his invention in the best way. It should be interpreted as meaning and concept consistent with the technical idea of the present invention based on the principle that there is.

According to the present invention, in a VR device that reproduces an image layer in which VR image content is displayed on a spherical curved surface, the image layer and the subtitle layer are separated by creating a subtitle layer with the same curvature as the image layer and overlaying it on the image layer. In spite of the existence of a subtitle layer, it is possible to improve the clarity and readability of the subtitle by exerting the effect that the subtitle layer is merged with the image layer.

In addition, since it is possible to detect a moving user's field of view and move the subtitle layer accordingly, it is possible to effectively deliver information about VR video content by preventing a situation in which the subtitle is cut or not visible within the user's field of view.

1 is a perspective view schematically illustrating a conventional virtual reality (VR) device.
2 is a block diagram of a device for processing captions for VR video content according to the first embodiment of the present invention.
3 is a cross-sectional view schematically illustrating a curved subtitle layer overlaid on an image layer by a subtitle processing device for VR image content according to the first embodiment of the present invention.
4 is a block diagram of a device for processing captions for VR video content according to a second embodiment of the present invention.
5 is a block diagram of a device for processing captions for VR video content according to a third embodiment of the present invention.
6 is a block diagram of a device for processing VR video content according to a fourth embodiment of the present invention.
7 is a block diagram of a caption processing apparatus for VR video content according to a fifth embodiment of the present invention.
8 is a block diagram of a device for processing VR video content according to a sixth embodiment of the present invention.
9 is a block diagram of a device for processing VR video content according to a seventh embodiment of the present invention.

The objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings and preferred embodiments. In the present specification, in adding reference numbers to the components of each drawing, it should be noted that only the same components are given the same number as possible even though they are indicated on different drawings. Also, terms such as “first” and “second” are used to distinguish one component from another, and the component is not limited by the terms. Hereinafter, in describing the present invention, detailed descriptions of related known technologies that may unnecessarily obscure the gist of the present invention will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a block diagram of an apparatus for processing VR video content according to a first embodiment of the present invention, and FIG. 3 is a diagram of an apparatus for processing VR video content overlaid on an image layer by the apparatus for processing VR video content according to the first embodiment of the present invention. It is a cross-sectional view schematically illustrating a curved caption layer.

As shown in FIGS. 2 to 3 , the device for processing VR video content according to the present invention includes: an image reproducing unit 10 that reproduces an image layer 11 in which VR video content is displayed on a virtual spherical curved surface; a subtitle layer generating unit 20 for generating a curved subtitle layer 21 in which a subtitle corresponding to the VR image content is displayed on a virtual curved surface curved with a predetermined curvature satisfying the following <Condition 1>; a subtitle reproducing unit 30 for reproducing the curved subtitle layer 21; a subtitle overlay unit 40 that overlays the reproduced curved subtitle layer 21 on the image layer 11; and a VR display unit 50 that simultaneously displays VR image content and subtitles on a VR screen.

<Condition 1>

Ki×80% ≤ Kc ≤ Ki×120%

Here, Ki is the curvature of the image layer 11 , and Kc is the curvature of the curved caption layer 21 .

The device for processing captions for VR video content according to the present invention is applied to a VR device that reproduces virtual reality (VR) video content and processes captions used for VR video content. When using as subtitles by writing , the subtitles are fixed at a predetermined position and the user's field of view is moved, and the subtitles are not visible. There is a problem in that sharpness and readability are poor and the fatigue of the eyes increases when watching for a long time, and the present invention has been devised as a solution to this problem.

The apparatus for processing VR video content according to the present invention includes an image reproduction unit 10 , a caption layer generation unit 20 , a subtitle reproduction unit 30 , a subtitle overlay unit 40 , and a VR display unit 50 . do.

The image reproducing unit 10 reproduces VR image content, wherein the VR image content is displayed on a virtual spherical curved surface and reproduced in the form of an image layer 11 . Here, the VR image content may be displayed on the entire or partial area of the virtual spherical curved surface.

The caption layer generator 20 generates the curved caption layer 21 . Here, the curved caption layer 21 is a layer in which a caption is displayed on a virtual curved surface curved with a predetermined curvature. In this case, the curvature of the curved caption layer 21 is determined to satisfy the following <condition 1> in relation to the curvature of the image layer 11 .

<Condition 1>

Ki×80% ≤ Kc ≤ Ki×120%

Here, Ki is the curvature of the image layer 11 , and Kc is the curvature of the curved caption layer 21 .

Preferably, the curvature of the curved layer 21 may be determined to be the same as the curvature of the image layer 11 . In this case, the curved caption layer 21 may overlap the entire surface of the image layer 11 without a gap.

Meanwhile, the caption of the curved caption layer 21 is a caption corresponding to VR image content. In this case, since the curved caption layer 21 is a virtual curved surface, the region except for the caption is made transparent, so that it does not affect the visual recognition of the overlapping image layer 11 . However, the region excluding the caption does not necessarily have to be transparent, and a predetermined color may be provided according to a user's selection or the like.

The subtitle reproducing unit 30 reproduces the curved subtitle layer 21 generated by the subtitle layer generating unit 20 . At this time, since the subtitle corresponds to the VR image content, the reproduction of the VR image content is sensed and the curved subtitle layer 21 corresponding thereto is reproduced.

The subtitle overlay unit 40 overlays the reproduced curved subtitle layer 21 on the matching image layer 11 . Here, the image layer 11 and the curved subtitle layer 21 are separate layers, but since their curvatures are similar or curved with the same curvature, the overlaid curved subtitle layer 21 is like an image layer. (11) improves the readability and clarity of subtitles by giving the same effect. Meanwhile, the above-described image reproducing unit 10, subtitle layer generating unit 20, subtitle reproducing unit 30, and subtitle overlay unit 40 may be implemented as a computing device in which software and hardware for performing information processing are combined. can

The VR display unit 50 is a device that simultaneously displays VR video content and subtitles on a VR screen. Here, the displayed VR image content is provided as the image layer 11 , and the subtitles are provided as the curved caption layer 21 . Meanwhile, the VR display unit 50 may be provided integrally with a VR device or may utilize a display of a smart terminal such as a smart phone or a tablet PC.

The subtitle processing device for VR video content according to the present invention can be applied to and used in a VR device that provides VR video content, and the above or later configurations are implemented with software, electrical and electronic devices, other hardware, etc., and can be applied to the VR device. In the case of a VR device provided or used by mounting a smart terminal, it may be mounted on the smart terminal. For example, the VR video content may be installed in a smart terminal in the form of a VR video content playback app and played by a user's selection. In addition, the caption labour generating unit may be mounted on a smart terminal that plays the VR video content in the form of a VR video content playback app.

Overall, according to the device for processing VR video content according to the present invention, in the VR device that reproduces the video layer 11 in which VR video content is displayed on a spherical curved surface, the caption layer has the same curvature as the video layer 11 . By creating 21 and overlaying it on the image layer 11 , the effect that the subtitle layer 21 is merged with the image layer 11 even though the image layer 11 and the subtitle layer 21 exist separately can improve the clarity and readability of subtitles.

4 is a block diagram of a device for processing captions for VR video content according to a second embodiment of the present invention.

Referring to FIG. 4 , the apparatus for processing VR video content according to the present invention may further include a caption storage unit 60 .

The caption storage unit 60 is a memory device for storing caption data including captions corresponding to VR image content, and provides captions to the caption layer generator 20 . Here, the caption storage unit 60 may include a flat caption layer in which the caption is displayed on a virtual plane. The flat subtitle layer has a problem in that the sharpness and readability of the subtitles decrease depending on the distance from the image layer 11 and increase eye fatigue during long exposure. The subtitle layer generator 20 converts the stored flat subtitle layer into an image layer. It can be transformed into the above-described curved caption layer 21 (refer to FIG. 3 ) by distorting the same as the curvature of (11). Accordingly, the caption data, which is provided in the form of a flat caption layer, is converted into the curved caption layer 21 and reproduced, thereby improving the clarity and readability of captions and eye fatigue.

5 is a block diagram of a device for processing captions for VR video content according to a third embodiment of the present invention.

As shown in FIG. 5 , the apparatus for processing VR image content according to the present invention may further include an image storage unit 70 .

The image storage unit 70 is a memory device for storing image data in which VR image content is recorded, and provides VR image content reproduced to the image playback unit 10 . Here, the image storage unit 70 may be implemented as the same or a separate element from the caption storage unit 60 .

On the other hand, the device for processing VR video content according to the present invention provides curved captions when captions are provided in a form in which text images are integrated into VR video content stored and reproduced in the video storage unit 70 . Transformation may be generated as a layer 21 (see FIG. 3 ). To this end, it may further include a text extraction unit 80 and a text data conversion unit 90 .

The text extraction unit 80 recognizes and extracts a text image from the VR image content stored in the image storage unit 70 . In this case, the text extraction unit 80 is configured to target the image layer 11 , a flat image constituting the image layer 11 before stitching, or a flat layer converted from the image layer 11 . , can recognize/extract text images. As an embodiment of the text extraction unit 80, an optical character recognition technology for recognizing characters by using an optical scanner and dedicated software to illuminate VR image content and convert the reflected beam into an electrical signal may be employed. However, the text extraction unit 80 is not necessarily limited to the optical character recognition technology, and may include any character reading device capable of recognizing and extracting text images from VR video content.

The text data conversion unit 90 converts the extracted text image into text data. The text data converted in this way is directly transmitted to the subtitle layer generating unit 20 or stored in the above-described subtitle storage unit 60 and then transmitted to the subtitle layer generating unit 20, and the subtitle layer generating unit 20 is transmitted. A curved caption layer 21 is generated based on the received text data.

6 is a block diagram of a device for processing VR video content according to a fourth embodiment of the present invention.

Referring to FIG. 6 , the apparatus for processing VR video content according to the present invention may further include a caption moving unit 100 .

The subtitle moving unit 100 moves the position of the reproduced curved subtitle layer based on the user's viewpoint. Hereinafter, an embodiment of the caption moving unit 100 will be described.

The subtitle moving unit 100 first identifies that VR video content is provided to the VR device, and identifies motion information (direction, speed, distance, etc.) that the VR device moves by moving the user's head, or detects the eyeball detection means. Through the location information of the user's pupil, the user's viewpoint is detected. Next, based on the detected user point of view, a recognizable range is generated centered on the viewing angle and the image point, and the curved caption layer 21 reproduced by the caption reproducing unit 30 in the generated recognizable range. By moving the position, the interaction between the reproduced VR image contents is matched, and a curved subtitle layer 21 that moves in real time is overlaid on the matched VR image content.

Therefore, the device for processing VR video content according to the present invention can detect a moving user's field of view and move the caption layer accordingly. information can be communicated effectively.

However, the subtitle moving unit 100 is not necessarily implemented according to the above-described method, and may be implemented through any known means as long as the curved subtitle layer 21 can be moved according to a user's viewpoint.

7 is a block diagram of a caption processing apparatus for VR video content according to a fifth embodiment of the present invention.

7 , the apparatus for processing VR video content according to the present invention may further include a caption option unit 110 .

The caption option unit 110 changes the shape of the caption displayed on the curved caption layer 21 . Here, the shape of the subtitle is size, position, color, font, kerning, length, margin, brightness, and line spacing, etc., and the subtitle option unit 110 can convert at least one of them according to the user's selection. have. By using the subtitle option unit 110, the user can enjoy a virtual experience by controlling the type of subtitle suitable for him/her.

8 is a block diagram of an apparatus for processing VR video contents according to a sixth embodiment of the present invention, and FIG. 9 is a block diagram of a device for processing captions for VR video contents according to a seventh exemplary embodiment of the present invention.

8 to 9 , the device for processing VR video content according to the present invention may provide information on VR video content according to a user's request. For example, when a user has a virtual experience of traveling on a travel destination through VR image content, there are times when he wants to know information about a specific building or architect, etc., but the subtitle processing device for VR image content according to the present invention is Information about it can be displayed in real time with subtitles. To implement this, the ROI identification unit 120 and the subtitle search unit 130 may be further included.

The ROI identification unit 120 identifies a predetermined ROI on the reproduced VR image content. Here, the region of interest means a region including an object of interest to the user among regions constituting the VR image content. The region of interest identification unit 120 may be implemented by employing, for example, a deep learning technique based on a convolutional neural network (CNN).

The caption search unit 130 searches for information on the region of interest corresponding to the case where the region of interest identification unit 120 identifies the region of interest. More specifically, the information on the object in the region of interest is acquired through the above-described subtitle storage unit 60 or Internet search, and the acquired data is transmitted to the subtitle layer generating unit 20, so that the subtitle layer generating unit (20) to create a curved caption layer 21 on which the acquired material is displayed.

Meanwhile, since the image layer 11 is displayed on a spherical curved surface and the user views the image while moving the field of view, the region of interest identification unit 120 may identify the region of interest only in the user's viewing angle range. That is, the ROI may be identified only when the ROI exists in the user's viewing angle range.

In addition, as shown in FIG. 9 , the apparatus for processing VR video content according to the present invention may further include a pupil detection unit 140 so that a user can select a region of interest in real time. Here, the pupil detection unit 140 detects the user's pupil position, detects the user's viewpoint, and measures the area to which the viewpoint is directed and the time the viewpoint stays. In this case, when the viewpoint is fixed within a predetermined region for a predetermined time, the region of interest identification unit 120 may identify the region of interest based on the user's viewpoint information. For example, when the user fixes a viewpoint for a specific building on the VR image content for 3 seconds or more, the eye detection unit 140 detects the user's viewpoint and transmits the detected viewpoint information to the ROI identification unit 120 . Then, the region of interest identification unit 120 identifies the specific building as the region of interest, and then the subtitle search unit 130 searches for information on the building, so that the subtitle layer generator 20 converts the information into a subtitle. indicate

Although the present invention has been described in detail through specific examples, this is for the purpose of describing the present invention in detail, and the present invention is not limited thereto, and by those of ordinary skill in the art within the technical spirit of the present invention. It is clear that the modification or improvement is possible.

All simple modifications and variations of the present invention belong to the scope of the present invention, and the specific scope of protection of the present invention will be made clear by the appended claims.

10: image playback unit 11: image layer
20: subtitle layer generator 21: curved subtitle layer
30: subtitle playback unit 40: subtitle overlay unit
50: VR display unit 60: subtitle storage unit
70: image storage unit 80: text extraction unit
90: text data conversion unit 100: subtitle moving unit
110: subtitle option unit 120: region of interest identification unit
130: subtitle search unit 140: pupil detection unit

Claims (10)

an image reproducing unit that reproduces an image layer in which VR image content is displayed on a virtual spherical curved surface;
a subtitle layer generator generating a curved subtitle layer in which a subtitle corresponding to the VR image content is displayed on a virtual curved surface curved with a predetermined curvature satisfying the following [Condition 1];
a subtitle reproducing unit for reproducing the curved subtitle layer;
a subtitle overlay unit that overlays the reproduced curved subtitle layer on the image layer;
a VR display unit that simultaneously displays the VR video content and the subtitles on a VR screen;
an ROI identifying unit for identifying a predetermined ROI on the reproduced VR image content; and
and a subtitle search unit configured to search the corresponding subtitle when the region of interest is identified, and transmit the retrieved subtitle to the subtitle layer generator.
<Condition 1>
Ki×80% ≤ Kc ≤ Ki×120%
Here, Ki is the curvature of the image layer, and Kc is the curvature of the curved caption layer.
The method according to claim 1,
a subtitle storage unit for storing the flat subtitle layer in which the subtitle is displayed on a virtual plane;
The caption layer generator is configured to convert the stored flat caption layer into the curved caption layer.
The method according to claim 1,
an image storage unit for storing the VR image content in which a text image is incorporated;
a text extraction unit for recognizing and extracting the text image from the stored VR image content; and
It further comprises; a character data conversion unit for converting the extracted text image into character data,
The caption layer unit is configured to generate the curved caption layer based on the converted text data.
The method according to claim 1,
and a subtitle moving unit for moving the reproduced curved subtitle layer to a position of the user's viewpoint.
The method according to claim 1,
A subtitle option unit for converting at least one of size, position, color, font, kerning, length, margin, brightness, and line spacing of the subtitle displayed on the curved subtitle layer; processing equipment.
The method according to claim 1,
The VR video content is a caption processing device for VR video content that is installed in a smart terminal in the form of a VR video content playback app and played by a user's selection.
The method according to claim 1,
The caption layer generating unit is a caption processing device for VR video content mounted on a smart terminal that plays the VR video content in the form of a VR video content playback app.
delete The method according to claim 1,
The region of interest identification unit may be configured to identify the region of interest when the region of interest exists in a user's viewing angle range.
The method according to claim 1,
Further comprising; a pupil detection unit for detecting the user's point of view by detecting the position of the user's pupil;
The region of interest identification unit may be configured to identify the region of interest based on the user's viewpoint information.

Images