JP2024116702A - Scene description editing device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a scene description editing device capable of properly editing a scene description defining a specific content in a case where a rendering function of a medium processing device is used by a user terminal.

SOLUTION: A scene description editing device comprises: an output unit which transmits, to a medium processing device, a scene description and viewpoint information defining the constitution of a content with a degree of freedom of a viewpoint; and an acquisition unit which acquires a specific content including at least the content, the specific content having been generated by the medium processing device based on the scene description and the viewpoint information.

SELECTED DRAWING: Figure 21

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to a scene description editing device.

Conventionally, mechanisms for transmitting content such as 360° video and 3D objects have been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which the user moves their head while in a seated position, and 6DoF, which involves viewpoint movement within the range in which the user can move freely. In such mechanisms, the positional relationship between the 360° video and the 3D objects is indicated by a scene description.

3GPP TR 26.928 V16.1.0 December 2020

In the above-mentioned context, a case can be considered in which specific content including content with viewpoint freedom is generated by a media processing device, and the generated specific content is then transmitted from the media processing device to a user terminal. Similarly, a case can be considered in which a scene description defining the specific content is edited by a scene description editing device.

After careful consideration, the inventors discovered that in the above-mentioned cases, it is necessary to share the rendering function of the media processing device between the user terminal and the scene description editing device.

The present invention has been made to solve the above-mentioned problems, and aims to provide a scene description editing device that makes it possible to appropriately edit scene descriptions that define specific content when the rendering function of a media processing device is used on a user terminal.

The outline of the disclosure is a scene description editing device that includes an output unit that outputs a scene description and viewpoint information that define the configuration of content having viewpoint freedom to a media processing device, and an acquisition unit that acquires specific content that includes at least the content and is generated by the media processing device based on the scene description and the viewpoint information.

The present invention provides a scene description editing device that enables appropriate editing of scene descriptions that define specific content when the rendering function of a media processing device is used on a user terminal.

FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. FIG. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the invention. FIG. 3 is a diagram for explaining the second content according to the embodiment. FIG. 4 is a diagram showing a method for viewing a specific content according to the embodiment. FIG. 5 is a diagram for explaining the first operation example. FIG. 6 is a diagram for explaining the second operation example. FIG. 7 is a diagram for explaining the second operation example. FIG. 8 is a diagram for explaining the third operation example. FIG. 9 is a diagram for explaining the third operation example. FIG. 10 is a diagram for explaining the operation example 3. In FIG. FIG. 11 is a diagram for explaining the operation example 3. In FIG. FIG. 12 is a diagram for explaining the fourth operation example. FIG. 13 is a diagram for explaining the fourth operation example. FIG. 14 is a diagram for explaining the fourth operation example. FIG. 15 is a diagram for explaining the fifth operational example. FIG. 16 is a diagram for explaining the fifth operational example. FIG. 17 is a diagram for explaining the fifth operational example. FIG. 18 is a diagram for explaining the fifth operational example. FIG. 19 is a diagram for explaining the first method according to the first modified example. FIG. 20 is a diagram for explaining the second method according to the first modified example. FIG. 21 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to the second modification. FIG. 22 is a diagram showing an example of a UI (User Interface) of a scene description editing device 600 according to the second modified example.

Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic, and the ratios of the dimensions may differ from the actual ones.

Therefore, specific dimensions should be determined with reference to the following explanation. Of course, there are also parts in which the dimensional relationships and ratios differ between the drawings.

[Disclosure Summary]
The scene description editing device according to the disclosed overview includes an output unit that outputs a scene description and viewpoint information that define the structure of content having viewpoint freedom to a media processing device, and an acquisition unit that acquires specific content that includes at least the content, the specific content being generated by the media processing device based on the scene description and the viewpoint information.

In the summary of the disclosure, the scene description editing device acquires specific content generated by the media processing device based on the scene description and the viewpoint information. With this configuration, assuming a case in which the specific content is generated (rendered) by the media processing device and then transmitted from the media processing device to a user terminal, the scene description editing device can acquire the specific content from the media processing device in a similar manner to that of the user terminal. Therefore, it is possible to appropriately edit the scene description while checking the specific content expected to be displayed on the user terminal.

It should be noted that the specific content generated by the media processing device is generated based on viewpoint information, and the images contained in the specific content can be treated as 2D images on the user terminal side.

[Embodiment]
(Transmission System)
A transmission system according to an embodiment will be described below. Fig. 1 is a diagram showing a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100, a media processing device 200, and a user terminal 300.

In an embodiment, the transmitting device 100 transmits a first content that does not have viewpoint freedom and a second content that has viewpoint freedom to the media processing device 200. Furthermore, the transmitting device 100 transmits first control information associated with the first content and second control information associated with the second content to the media processing device 200.

The first content may include at least one of 2D video and audio. The first content and the first control information may be transmitted in a first method. The first method may be a method conforming to ISO/IEC 23008-1 (hereinafter, MMT (MPEG Media Transport)). In the following, a case will be illustrated in which the first method is MMTP (MMT Protocol) conforming to MMT. The first control information may be referred to as MMT-SI (Signaling Information).

The second content may include a 360° video and a 3D object. The second content and the second control information may be transmitted in a second manner. Alternatively, they may be transmitted in a protocol such as HTTP (Hyper Text Transfer Protocol). The second content may conform to 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which the user moves his/her head while in a sitting position, or 6DoF, which involves viewpoint movement within the range in which the user can move freely. Since the second content has viewpoint freedom, it may include two or more 360° videos and two or more 3D objects at the same time (frame). The second control information may be referred to as a scene description.

Here, the second control information may be transmitted in the first method described above. That is, the second control information may be transmitted in the same first method (e.g., MMTP) as the first control information. Alternatively, the second control information may be transmitted in a protocol such as HTTP.

The transmission from the transmitting device 100 to the media processing device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communication network.

Although not limited thereto, the transmission system may be a digital wireless transmission system. The digital wireless transmission system may be a system used for 4K and 8K satellite broadcasting.

The media processing device 200 generates specific content including at least the above-mentioned second content based on the viewpoint information received from the user terminal 300, and transmits the generated specific content to the user terminal 300. Although not particularly limited, the transmission of the specific content may be transmission using the Internet network or transmission using a mobile communication network.

The user terminal 300 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display. As shown in FIG. 1, two or more user terminals 300 may be provided as the user terminal 300. In other words, two or more user terminals 300 may request the media processing device 200 to generate specific content. Each user terminal 300 may transmit different viewpoint information to the media processing device 200.

(Media processing device and user terminal)
A media processing device and a user terminal according to an embodiment of the present invention will now be described. Fig. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the present invention.

First, the media processing device 200 has a reception unit 210, a renderer 220, and an encoding processing unit 230.

The reception unit 210 receives viewpoint information. In an embodiment, the reception unit 210 constitutes a receiving unit that receives viewpoint information from the user terminal 300. The viewpoint information includes an information element indicating the viewpoint position of the user of the user terminal 300 and an information element indicating the line of sight of the user of the user terminal 300.

The renderer 220 generates specific content including at least the second content based on the viewpoint information. Since the specific content is generated based on the viewpoint information, it may include one 360° video and one 3D object at the same time (frame). Below, an example is given of the specific content including the first content in addition to the second content.

First, as shown in FIG. 2, the renderer 220 generates a first content including 2D video and audio as part of a specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.

Specifically, the renderer 220 acquires the 2D video, audio, and MMT-SI in the form of an MMTP packet in which the 2D video, audio, and MMT-SI are packetized.

For example, MMTP packets are stored in IP (Internet Protocol) packets. IP packets may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).

Here, the first content is processed in units (hereinafter, MPU: Media Processing Unit) that are separated by a certain time width. The MPU includes one or more access units. The access unit may be treated as an MFU (Media Fragment Unit). The MFU related to 2D video may be called a NAL (Network Abstraction Layer) unit, and the MFU related to audio may be called an MHAS (MPEG-H 3D Audio Stream) packet.

The MMT-SI includes a PA (Package Access) message, and the PA message includes an MPT (MMT Package Table) that indicates a list of the first content. Furthermore, the MMT-SI includes an MPU timestamp descriptor that indicates the presentation time of the first content. The MPU timestamp descriptor may mean the presentation time of the MPU, i.e., the time of the access unit that is first presented in the MPU.

The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or may be a time provided by GPS (Global Positioning System). The reference time may be a time provided by an NTP (Network Time Protocol) server or a PTP (Precision Time Protocol) server.

Secondly, as shown in FIG. 2, the renderer 220 generates second content including a 360° video and a 3D object as part of the specific content based on the second control information (scene description). The viewpoint information is used in generating the second content.

Specifically, the renderer 220 may acquire the scene description in the form of an MMTP packet in which the scene description is packetized. There are no particular limitations on the method of acquiring the 360° video and 3D objects.

The 360° video may be converted to 2D video by a projective transformation such as ERP (Equirectangular projection) or cube mapping. Metadata indicating the type of projective transformation applied to the 360° video may be added. The 3D object may be encoded in a mesh format. ISO/IEC 14496-16 "Animation framework extension (AFX)" may be used for encoding the mesh format. The 3D object may be encoded in a point cloud format. ISO/IEC 23090-5 "Video-based Point Cloud Compression" may be used for encoding the point cloud format.

Here, the second content is compiled into one file in units separated by a certain time width. The certain time width may be 500 ms. For example, if the frame rate is 60 fps (frames per second), one file contains 30 frames.

The scene description is generated for each file and includes information for each frame that identifies the 360° video and 3D objects. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, an information element (scale_object) indicating the size of the 3D object in the frame, etc.

Third, the renderer 220 outputs the specific content including the first content and the second content to the encoding processing unit 230. The renderer 220 may output the presentation time of the specific content together with the specific content to the encoding processing unit 230.

Here, the presentation time of the specific content may be corrected based on the delay time between the media processing device 200 and the user terminal 300. Specifically, the renderer 220 may calculate the presentation time (T' = T + ΔT) of the specific content provided from the media processing device 200 to the user terminal 300 based on the presentation time (T) and delay time (ΔT) of the specific content provided from the transmission device 100 to the media processing device 200. The delay time (ΔT) may be a value that is predetermined in the media processing device 200, or may be a different value for each user terminal 300.

Fourthly, the renderer 220 may constitute a transmission unit that transmits the viewpoint information used to generate the specific content to the user terminal 300. The viewpoint information used to generate the specific content may be transmitted from the encoding processing unit 230 to the user terminal 300.

For example, the transmission method of the viewpoint information and the specific content may be MMTP or HTTP. When MMTP is used as the transmission method of the specific content, the viewpoint information may be stored as metadata in OMAF (Omnidirectional Media Format) defined in ISO/IEC 23090-2.

The encoding processing unit 230 encodes the specific content generated by the renderer 220. In an embodiment, the encoding processing unit 230 may be an example of a transmission unit that transmits the specific content to the user terminal 300.

Furthermore, the encoding processing unit 230 may encode the presentation time of the specific content. The encoding processing unit 230 may transmit an information element indicating the presentation time together with the specific content to the user terminal 300.

Here, the encoding processing unit 230 can use any compression encoding method as the compression encoding method. For example, the compression encoding method may be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

As described above, the second content included in the specific content is generated based on viewpoint information, so the video included in the specific content can be treated as a 2D video that does not have freedom of viewpoint.

For example, the transmission control method used to start and end viewing of specific content may include RTSP (Real Time Streaming Protocol). The transmission method may be MMTP or HTTP. When MMTP is used as the transmission method, the specific content may be stored in OMAF defined in ISO/IEC 23090-2.

As shown in FIG. 2, the user terminal 300 has a detection unit 310, a decoding processing unit 320, and a renderer 330.

The detection unit 310 detects the user's viewpoint position and line of sight direction. The detection unit 310 may include an acceleration sensor or a GPS (Global Positioning System) sensor. The detection unit 310 may include a user I/F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) that is manually input by the user. The detection unit 310 may transmit viewpoint information (viewpoint position and line of sight direction) to the media processing device 200. The detection unit 310 may output the viewpoint information (viewport) to the renderer 330.

The decoding processing unit 320 decodes the specific content received from the media processing device 200. The decoding processing unit 320 may decode the presentation time received from the media processing device 200. The decoding processing unit 320 may output the specific content to the renderer 330, or may output the presentation time to the renderer 330.

The renderer 330 outputs the specific content decoded by the decoding processing unit 320. The renderer 330 may output the specific content based on the presentation time decoded by the decoding processing unit 320. For example, the renderer 330 may output the video content included in the specific content to a display and output the audio content included in the specific content to a speaker.

Here, the renderer 330 may generate specific content in which the viewpoint position and line of sight direction are corrected based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.

(Second content)
The second content according to the embodiment will be described below. Here, the second content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.

For example, as shown in FIG. 3, at t=0, a 360° video may be displayed without a 3D object being displayed. The 360° video may be considered to be a background video of the 3D object. At t=1, a 3D object may be displayed in a form superimposed on the 360° video. Furthermore, at t=1, the position and rotation of the 3D object superimposed on the 360° video may be changed.

The above-mentioned scene description includes information elements that indicate the position, rotation, and size of the 3D object for each of t=0, t=1, and t=2, and allows the 3D object to be appropriately superimposed on the 360° video.

(How to watch)
A viewing method according to the embodiment will be described below, in which viewing of a specific content including a first content and a second content will be exemplified.

As shown in FIG. 4, in step S11, the user terminal 300 sends an RTSP SETUP to the media processing device. The RTSP SETUP is a message indicating that viewing of a specific content is to begin.

Here, the RTSP SETUP includes the IP address of the user terminal 300, the listening port number, the content identification information (content ID), etc. The RTSP SETUP may also include capability information of the user terminal 300 for viewing specific content. The capability information may include the frame rate, the display resolution, etc. The display resolution may include the field of view (FoV). The capability information may also include information elements indicating the encoding method and compression method supported by the user terminal 300.

Here, a case is illustrated in which the capability information of the user terminal 300 is directly notified to the media processing device 200, but the embodiment is not limited to this. The capability information of the user terminal 300 may be notified to the transmitting device 100, and then notified from the transmitting device 100 to the media processing device 200.

In step S12, the media processing device 200 transmits a response to the RTSP SETUP. Here, an ACK is transmitted as a response indicating that the RTSP SETUP has been accepted.

In step S21, the user terminal 300 transmits the initial viewpoint information to the media processing device 200. The initial viewpoint information may be transmitted in the MMT-SI format.

In step S22, the media processing device 200 generates initial specific content based on the initial viewpoint information (rendering process). For example, the media processing device 200 generates second content to be included in the initial specific content based on the initial viewpoint information and the scene description.

Here, the media processing device 200 may generate the initial specific content using a range wider than the display resolution of the user terminal 300 as the viewport. For example, the range wider than the display resolution may be a range of the display resolution + 20% in the horizontal direction and the display resolution + 20% in the vertical direction.

The media processing device 200 applies a compression encoding method to the initial specific content. Although not particularly limited, the compression encoding method may be HEVC or VVC.

In step S23, the media processing device 200 transmits the initial specific content corresponding to the initial viewpoint information to the user terminal 300. The media processing device 200 transmits the presentation time of the initial specific content to the user terminal 300. As described above, the presentation time (T') provided to the user terminal 300 may be determined based on the delay time (ΔT).

When a different value is used for the delay time (ΔT) for each user terminal 300, it is possible to specify it on the media processing device 200 side by including the transmission time of the RTSP SETUP in the above-mentioned RTSP SETUP.

The user terminal 300 outputs the specific content based on the presentation time (T'). The user terminal 300 may generate the specific content with the viewpoint position and line of sight corrected based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.

In step S31, the user terminal 300 transmits viewpoint information to the media processing device 200. The viewpoint information may be transmitted in the format of MMT-SI. Here, the user terminal 300 may transmit the viewpoint information at a predetermined period (e.g., 500 ms), or may transmit the viewpoint information in response to a change in at least one of the viewpoint position and the line of sight direction.

In step S32, the media processing device 200 generates specific content based on the viewpoint information received in step S31 (rendering process).

In step S33, the media processing device 200 transmits to the user terminal 300 the specific content that corresponds to the viewpoint information received in step S31.

The processing of steps S31 to S33 is similar to the processing of steps S21 to S23, except that the viewpoint information received in step S31 is used instead of the initial viewpoint information. Therefore, details of the processing of steps S31 to S33 are omitted. The processing of steps S31 to S33 may be repeated at a predetermined interval, or may be repeated every time the user's viewpoint position or line of sight changes.

In step S41, the user terminal 300 sends an RTSP TEARDOWN to the media processing device. The RTSP TEARDOWN is a message indicating that viewing of a specific content is to end.

In step S42, the media processing device 200 transmits a response to the RTSP TEARDOWN. Here, an ACK is transmitted as the response, indicating that the RTSP TEARDOWN has been accepted.

In FIG. 4, a case where steps S11 and S12 are performed on an RTSP basis is illustrated, but the embodiment is not limited to this. Steps S11 and S12 may be performed on an MMTP basis or on an HTTP basis.

Similarly, although the example shows a case where steps S41 and S42 are performed on an RTSP basis, the embodiment is not limited to this. Steps S41 and S42 may be performed on an MMTP basis or on an HTTP basis.

In FIG. 4, a case where steps S31 to S33 are executed based on MMTP is illustrated, but the embodiment is not limited to this. Steps S31 to S33 may be executed based on another method (e.g., HTTP).

Similarly, although the example shows a case where steps S41 to S43 are executed based on MMTP, the embodiment is not limited to this. Steps S41 to S43 may be executed based on other methods (e.g., HTTP).

(Example 1)
The above-described embodiment may include the following Operation Example 1. In Operation Example 1, the media processing device 200 transmits to the user terminal 300 viewpoint information used in generating the specific content, in association with a sequence number added to the specific content.

Specifically, the media processing device 200 (renderer 220) generates second content including a 360° video and a 3D object as part of a specific content based on the second control information (scene description), as in the above-described embodiment. In generating the second content, viewpoint information received from the user terminal 300 is used.

In operation example 1, the renderer 220 associates the viewpoint information used in generating the specific content (here, the second content) with a sequence number. The renderer 220 transmits the viewpoint information used in generating the specific content to the user terminal 300 in association with the sequence number added to the specific content. The viewpoint information may be stored in a VP (View Port) message. The VP message may have the format of MMT-SI defined in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc. The VP message may be transmitted for each frame. The VP message may be transmitted to the user terminal 300 as a message related to MMTP (MMT-SI).

Although not limited to this, the VP message may have the data structure shown in FIG. 5. As shown in FIG. 5, the VP message may include message_id, version, length, fov, viewpoint_pos_x, viewpoint_pos_y, viewpoint_pos_z, viewpoint_yaw, viewpoint_pitch, viewpoint_roll, viewport_width, viewport_height, mpu_sequence_number_flag, mpu_sequence_number, etc.

The message_id is identification information that indicates the VP message. The message_id may be 0x0204.

The version is information that indicates the version of the MMTP protocol. The version may be 0x00.

Length is information indicating the length of the VP message.

fov is information that indicates the field of view.

viewpoint_pos_x is information that indicates the x coordinate of the viewpoint position. viewpoint_pos_x is an example of viewpoint information used in generating specific content.

viewpoint_pos_y is information that indicates the y coordinate of the viewpoint position. viewpoint_pos_y is an example of viewpoint information used in generating specific content.

viewpoint_pos_z is information that indicates the z coordinate of the viewpoint position. viewpoint_pos_z is an example of viewpoint information used in generating specific content.

viewpoint_yaw is information that indicates the yaw of the viewpoint position. viewpoint_yaw is an example of viewpoint information used in generating specific content.

viewpoint_pitch is information that indicates the pitch of the viewpoint position. viewpoint_pitch is an example of viewpoint information used in generating specific content.

viewpoint_roll is information that indicates the role of the viewpoint position. viewpoint_roll is an example of viewpoint information used in generating specific content.

viewport_width is information that indicates the width of the display area (specific content).

viewport_height is information that indicates the height of the display area (specific content).

The mpu_sequence_number_flag is information indicating whether or not the mpu_sequence_number field exists. For example, if the mpu_sequence_number_flag is 1, the mpu_sequence_number field exists, and if the mpu_sequence_number_flag is 0, the mpu_sequence_number field does not have to exist.

The mpu_sequence_number is the MPU sequence number of the video corresponding to the specific content indicated by the VP message. The mpu_sequence_number is an example of a sequence number added to specific content.

Here, viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z are examples of information elements that indicate the user's viewpoint position in the three-dimensional space configured by the scene description. viewpoint_yaw, viewpoint_pitch, and viewpoint_roll are examples of information elements that indicate the user's line of sight in the three-dimensional space configured by the scene description. viewport_width and viewport_height are examples of information elements that indicate the number of pixels of an image included in specific content.

Under these conditions, the media processing device 200 and the user terminal 300 may perform the following operations.

First, the media processing device 200 (renderer 220) may determine viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z based on the viewpoint position used to generate the specific content. The renderer 220 may determine viewpoint_yaw, viewpoint_pitch, and viewpoint_roll based on the line of sight direction used to generate the specific content. The renderer 220 may determine viewport_width based on the number of pixels in the horizontal direction of the specific content, and may determine viewport_height based on the number of pixels in the vertical direction of the specific content.

The media processing device 200 (encoding processing unit 230) may perform compression encoding of the specific content generated by the renderer 220, and transmit the specific content. Here, the encoding processing unit 230 may transmit the VP message shown in FIG. 5 to the user terminal 300. In other words, the encoding processing unit 230 may transmit the viewpoint information used in generating the specific content to the user terminal 300, in association with the sequence number added to the specific content.

Secondly, the user terminal 300 (decoding processing unit 320) may constitute a receiving unit that receives viewpoint information used in generating a specific content from the media processing device 200 in association with a sequence number added to the specific content. That is, the decoding processing unit 320 may receive the VP message (MMT-SI) shown in FIG. 5 from the media processing device 200.

The user terminal 300 (renderer 330) may associate the video constituting the decoded specific content with the viewpoint information included in the VP message based on the sequence number (mpu_sequence_number). The renderer 330 may identify a video in a range identified by the viewpoint information detected by the detection unit 310 from the display area defined by the information included in the VP message (viewport_width and viewport_height) based on the difference between the viewpoint information detected by the detection unit 310 and the viewpoint information included in the VP message. The renderer 330 may display the identified video.

(Example 2)
The above-described embodiment may include the following operation example 2. In the operation example 2, as shown in FIG. 6, a case is assumed in which a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed. The first user terminal 400 may be a terminal such as a head-mounted display. The first user terminal may have the same function as the above-described user terminal 300. The second user terminal 500 may be a terminal such as a volumetric display.

Specifically, in the second operation example, as shown in FIG. 6, the media processing device 200 (renderer 220) may constitute a receiving unit that receives the content configuration and the recommended viewport information from the transmitting device 100. The content configuration may be considered to include 2D video, audio, 360° video, and 3D objects. The content configuration may be considered to include MMT-SI and a scene description. The recommended viewport information may be considered to be an example of specific viewpoint information. The recommended viewport information may be information that defines (recommends) the position, direction, and angle of view at which the video is viewed in a three-dimensional space (a three-dimensional space constructed by a scene description) constituted by the specific content. The recommended viewport information may include at least one of an information element indicating the viewpoint position in the three-dimensional space constituted by the specific content and an information element indicating the line of sight direction in the three-dimensional space constituted by the specific content. The recommended viewport information may be considered to be viewpoint information used primarily in the second user terminal 500.

In the following, unless explicitly stated otherwise, the viewpoint information used in generating specific content may include specific viewpoint information (recommended viewport information) received from the transmitting device 100, or may include viewpoint information received from the user terminal 300.

Although not limited thereto, the recommended viewport information may be included in the scene description in the manner shown in FIG. 7. As shown in FIG. 7, the recommended viewport information may include camera_orientation, frame_number, translation, and yfov.

Camera_orientation is information that indicates the direction in which the image is viewed in the three-dimensional space constructed by the scene description. Camera_orientation can be considered synonymous with line of sight.

frame_number is information that indicates the frame number of the image to which camera_orientation, translation, and yfov are applied. Camera_orientation may be considered an example of specific viewpoint information.

Translation is information that indicates the position from which an image is viewed in the three-dimensional space constructed by the scene description. Translation may be considered to be synonymous with viewpoint position. Translation may be considered to be an example of specific viewpoint information. For example, Figure 7 illustrates a case in which the viewpoint position is [0,0,-50] when the frame number is 0, and the viewpoint position moves to [0,0,-75] when the frame number is 2505.

yfov is information that indicates the angle of view from which the image is viewed in the three-dimensional space constructed by the scene description.

Although not limited thereto, camera_orientation and translation may be assigned by the creator of the content. Alternatively, in the case where the content is captured by a camera, camera_orientation and translation may be assigned automatically by the GPS and sensors installed in the camera.

Here, translation is an example of an information element that indicates the viewpoint position in the three-dimensional space formed by a specific content. camera_orientation is an example of an information element that indicates the line of sight direction in the three-dimensional space formed by a specific content.

Under these conditions, the media processing device 200 and the user terminal 300 may perform the following operations.

First, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information). The media processing device 200 (encoding processing unit 230) may perform compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding processing unit 230 may transmit the specific content generated based on the specific viewpoint information to the first user terminal 400, or may transmit the specific content generated based on the specific viewpoint information to the second user terminal 500.

Secondly, the media processing device 200 (reception unit 210) may receive viewpoint information from the first user terminal 400 when the user terminal is the first user terminal 400 that feeds back viewpoint information. The media processing device (renderer 220) may generate specific content based on the viewpoint information received from the first user terminal 400. In such a case, the media processing device 200 (reception unit 210) may receive a reset signal from the first user terminal 400. The media processing device (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information) in response to the reset signal.

In such a case, the first user terminal 400 may have a configuration similar to that of the user terminal 300. However, the first user terminal 400 (detection unit 310) may have a function for detecting a reset signal. The detection unit 310 may detect a user operation to input a reset signal. The detection unit 310 may transmit the reset signal to the media processing device 200.

Thirdly, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information) when the user terminal is a second user terminal 500 that does not feed back viewpoint information.

In such a case, the second user terminal 500 may not have a detection unit 310 that detects viewpoint information. The second user terminal 500 may not have a renderer 330. The second user terminal 500 may have the same configuration as the user terminal 300, except that it does not have the detection unit 310 and the renderer 330.

(Example 3)
The above-described embodiment may include the following Operation Example 3. In Operation Example 3, media processing device 200 (e.g., selection unit 260 described later) may constitute a receiving unit that receives, from transmitting device 100, quality information on each of two or more streams whose quality differs depending on the orientation of the 3D object based on viewpoint information, as quality information on a stream related to a 3D object included in specific content.

Specifically, in operation example 3, as shown in FIG. 8, the media processing device 200 has a selection unit 260 in addition to the configuration shown in FIG. 2. The selection unit 260 receives a scene description and a 3D object from the transmitting device 100. The selection unit 260 inputs a stream (3D object) selected from two or more streams to the renderer 220. The selection unit 260 may request the transmitting device 100 to transmit the selected stream. Note that, in a case in which the media processing device 200 transmits specific content to multiple user terminals 300, the selection unit 260 may request the transmitting device 100 to transmit a stream required by each of the multiple user terminals 300, or may request the transmitting device 100 to transmit all streams.

Here, the selection unit 260 receives, from the transmission device 100, quality information regarding each of two or more streams whose quality differs depending on the orientation of the 3D object based on the viewpoint information, as quality information of the stream regarding the 3D object.

The quality information may be information indicating the relative quality of each face constituting a bounding box for a 3D object. The bounding box may be represented by a three-dimensional rectangle onto which the 3D object is projected. For example, as shown in FIG. 9, the bounding box may be defined by vertices A to H. In such a case, the faces of the bounding box include face #1 represented by vertices A, B, F, E, face #2 represented by vertices B, C, G, F, face #3 represented by vertices A, B, C, D, face #4 represented by vertices E, F, G, H, face #5 represented by vertices A, D, H, E, and face #6 represented by vertices D, C, G, H.

In such a case, when we consider the case of viewing a 3D object in the 3D space constructed by the scene description, we assume that three faces are primarily observed. In other words, we assume that the remaining three faces are not observed very often.

In operation example 3, two or more streams are prepared as streams related to 3D objects included in specific content, with the quality differing depending on the orientation of the 3D object based on viewpoint information.

Although not limited to this, the quality information may be included in the scene description in the manner shown in FIG. 10. In FIG. 10, six streams are illustrated as streams with different orientations of 3D objects. The quality information may be expressed in the form of "quality" [#1,#2,#3,#4,#5,#6]. In the [ ], #1 to #6 refer to the quality indexes of surfaces #1 to #6. The quality index may take values in the range of 1 to 9. The higher the quality index value, the higher the quality. For example, in a stream identified by "id"="1", the quality ("8") of #1, #2, and #3 is high, and the quality ("3") of #4, #5, and #6 is low. In a stream identified by "id"="2", the quality ("3") of #1, #2, and #3 is low, and the quality ("8") of #4, #5, and #6 is high.

Under these conditions, the media processing device 200 may perform the following operations. In the following, the selection of a stream (3D object) selected from two or more streams is mainly described.

In mode 1, as shown in the upper part of FIG. 11, the media processing device 200 (selection unit 260) may identify the vertex closest to the user's viewpoint (for example, vertex B), and then identify the three faces that have the closest vertex (for example, face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). The selection unit 260 may select the stream that maximizes the sum of the quality indexes of the three identified faces (in the example shown in FIG. 10, the stream identified by "id"="1").

In addition, in mode 1, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and quality information.

Mode 2 may be a mode applied in cases where a reduced or enlarged display of a 3D object is performed. For example, as shown in the middle of FIG. 11, the media processing device 200 (selection unit 260) may identify the vertex (e.g., vertex B) closest to the user's viewpoint position, and then identify three faces having the closest vertex (e.g., face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). In cases where a reduced display of a 3D object is performed, pixels of the 3D object are thinned out. Therefore, the selection unit 260 may select a stream (in the example shown in FIG. 10, the stream identified by "id"="2") that minimizes the sum of the quality indexes of the three identified faces. On the other hand, in cases where an enlarged display of a 3D object is performed, pixels of the 3D object are interpolated. Therefore, the selection unit 260 may select a stream (in the example shown in FIG. 10, the stream identified by "id"="1") that maximizes the sum of the quality indexes of the three identified faces.

In addition, in mode 2, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and quality information.

Mode 3 may be applied when two 3D objects (3D object #1 and 3D object #2) overlap in the user's line of sight. Here, the selection of the stream related to 3D object #1 will be described. For example, as shown in the lower part of FIG. 11, the media processing device 200 (selection unit 260) may identify the vertex (e.g., vertex B) closest to the user's viewpoint position, and then identify three faces having the closest vertex (e.g., face #1 represented by vertices A, B, F, and E, face #2 represented by vertices B, C, G, and F, and face #3 represented by vertices A, B, C, and D). Here, 3D object #2 overlaps on the line segment connecting the vertex (e.g., vertex B) closest to the user's viewpoint position and the user's viewpoint position, and the identified three faces are blocked by 3D object #2. Therefore, the selection unit 260 may select the stream (in the example shown in FIG. 10, the stream identified by "id"="2") that has the smallest sum of the quality indexes of the identified three faces.

In addition, in mode 3, since the vertex closest to the user's viewpoint position is identified based on the viewpoint information, the selection unit 260 may be considered to select the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information and quality information. Furthermore, in mode 3, since the overlap of two 3D objects is identified based on the viewpoint information and the arrangement information of the 3D objects, the selection unit 260 may be considered to select the stream to be transmitted to the user terminal 300 from among two or more streams based on the viewpoint information, quality information, and arrangement information. The arrangement information of the 3D objects (for example, "rotation_object", "scale_object", and "translation_object" shown in FIG. 10) may be included in the scene description. The arrangement information of the 3D objects may be considered to be "link_area" shown in FIG. 10. That is, the selection unit 260 may receive the arrangement information of the 3D objects in the three-dimensional space from the transmission device 100.

Note that in the quality information shown in FIG. 10, the sum of the quality indexes of the six aspects is equal in each stream. However, the embodiment is not limited to this. The sum of the quality indexes of the six aspects may be different between two or more streams.

(Example 4)
The above-described embodiment may include the following Operation Example 4. Here, Operation Example 4 includes the operations described below in addition to Operation Example 3. In Operation Example 4, media processing device 200 (e.g., selection unit 270 described below) may constitute a receiving unit that receives importance information related to each of two or more objects included in specific content from transmitting device 100.

Specifically, in operation example 4, as shown in FIG. 12, the media processing device 200 has a selection unit 270 in addition to the configuration shown in FIG. 2. The selection unit 270 receives a scene description, a 3D object, and a 360° video from the transmission device 100. The selection unit 270 inputs a stream (3D object) selected from two or more streams to the renderer 220.

Here, the selection unit 270 receives importance information regarding each of the two or more objects from the transmission device 100. The objects may include 3D objects and 360° video.

For example, the importance information may be information indicating the relative importance between two or more objects. For example, as shown in FIG. 13, consider a case in which object A (background), object B (person), and object C (dog) exist in a three-dimensional space constructed by a scene description. Object A (background) is an example of a 360° image, and object B (person) and object C (dog) are examples of 3D objects. In such a case, the importance information may be information indicating the relative importance between each of object A (background), object B (person), and object C (dog).

Although not particularly limited, the importance information may be included in the scene description in the manner shown in FIG. 14. In FIG. 14, the importance information may be represented by weight. Weight may take a value ranging from 1 to 9. A larger weight value may indicate a higher importance. FIG. 14 illustrates a case in which object A (background) identified by "object_id"="0" has the highest weight ("9"), object B (person) identified by "object_id"="1" has the lowest weight ("3"), and object C (dog) identified by "object_id"="2" has a weight ("8") higher than the weight of object B (person) and lower than the weight of object A (background).

Under these assumptions, the media processing device 200 may execute the following operations. In the following, the selection of a stream (3D object) selected from two or more streams is mainly described.

First, the media processing device 200 (selection unit 270) selects the stream with the highest quality for the 3D object with the highest importance. The stream selection method may be the same as in operation example 3. For example, because the importance of object C (dog) is greater than the importance of object B (person), the selection unit 270 selects the stream for object C (dog) that maximizes the sum of the quality indexes of the three faces that have the vertex closest to the user's viewpoint.

Secondly, the media processing device 200 (selection unit 270) selects the lowest quality stream for 3D objects other than the 3D object with the highest importance. Next, the selection unit 270 replaces the lowest quality stream with a higher quality stream within the range in which the specific condition is satisfied, starting from the 3D object with the highest importance. The specific condition may include a first condition that the bandwidth of the line from the transmission device 100 to the media processing device 200 is equal to or lower than a threshold, and may include a second condition that the processing load of the media processing device 200 is equal to or lower than a threshold. The specific condition may be defined by a combination of the first and second conditions. For example, because the importance of object B (person) is lower than the importance of object C (dog), the selection unit 270 selects a high quality stream for object B (person) within the range in which the specific condition is satisfied.

As described above, the media processing device 200 (selection unit 270) may be considered to select a stream to be transmitted to the user terminal 300 from among two or more streams based on viewpoint information, quality information, and importance information. The media processing device 200 (selection unit 270) may be considered to select a stream to be transmitted to the user terminal 300 from among two or more streams based on viewpoint information, quality information, placement information, and importance information.

Note that in operation example 4, a case where one stream exists for 360° video has been exemplified. However, the embodiment is not limited to this. For 360° video, two or more streams of different qualities may also exist.

In the fourth operational example, a case has been exemplified in which, for a 3D object, there are two or more streams whose quality differs depending on the orientation of the 3D object based on viewpoint information. However, the embodiment is not limited to this. For a 3D object, there may be two or more streams whose quality differs regardless of the orientation of the 3D object.

(Example 5)
The above-described embodiment may include the following Operation Example 3. In Operation Example 3, the media processing device 200 (e.g., the renderer 220) may constitute a receiving unit that receives, from the transmitting device 100, an information element that defines a movement range of a user's viewpoint position in a three-dimensional space configured by specific content (a three-dimensional space constructed by a scene description).

First, in operation example 5, the information element may include an information element (hereinafter, a first information element) that restricts movement of the user's viewpoint position to inside a 3D object included in specific content. For example, as shown in FIG. 15, in a case where a 3D object is placed in a three-dimensional space constructed by a scene description, movement of the viewpoint position to inside the 3D object may be restricted. However, in a case where the 3D object is a building or another scene exists inside the 3D object, movement of the viewpoint position to inside the 3D object may be permitted.

Secondly, in operation example 5, the information element may include an information element (hereinafter, second information element) that restricts movement of the user's viewpoint position outside the three-dimensional space. For example, as shown in FIG. 16, the three-dimensional space may be defined by a combination of a rectangular parallelepiped and a spheroid. The number of rectangular parallelepipeds that define the three-dimensional space may be two or more, and the number of spheroids that define the three-dimensional space may be two or more. However, there may be cases in which movement of the user's viewpoint position outside the three-dimensional space is permitted.

Although not particularly limited, the first information element may be included in the scene description in the manner shown in FIG. 17. In FIG. 17, the first information element may be represented by viewing_inside_object_flag. The viewing_inside_object_flag may be set for each 3D object. For example, when viewing_inside_object_flag is "0", movement of the viewpoint position into the 3D object may be restricted, and when viewing_inside_object_flag is "1", movement of the viewpoint position into the 3D object may be permitted.

Although not particularly limited, the second information element may be included in the scene description in the manner shown in FIG. 18. In FIG. 18, the second information element may include information elements (cuboid_center_x, cuboid_center_y, cuboid_center_z, cuboid_size_x, cuboid_size_y, cuboid_size_z) that define a rectangular parallelepiped that defines a three-dimensional space. The cuboid_center_x, cuboid_center_y, and cuboid_center_z are information elements that indicate the center position of the rectangular parallelepiped, and the cuboid_size_x, cuboid_size_y, and cuboid_size_z are information elements that indicate the size of the rectangular parallelepiped. The second information element may include information elements (spheroid_center_x, spheroid_center_y, spheroid_center_z, spheroid_size_x, spheroid_size_y, spheroid_size_z) that define a spheroid that defines a three-dimensional space. spheroid_center_x, spheroid_center_y, and spheroid_center_z are information elements that indicate the center position of a spheroid, and spheroid_size_x, spheroid_size_y, and spheroid_size_z are information elements that indicate the size of the spheroid. Note that cuboid_enable is an information element that indicates whether or not a three-dimensional space is defined by a rectangular parallelepiped, and spheroid_enable may be an information element that indicates whether or not a three-dimensional space is defined by a spheroid. FIG. 18 illustrates an example in which a three-dimensional space is defined by two rectangular parallelepipeds and two spheroids.

Under these assumptions, the media processing device 200 (renderer 220) may perform the operations described below.

First, when the user's viewpoint position moves outside the movement range, the renderer 220 may generate specific content by using the intersection point of the trajectory of the user's viewpoint position and the boundary of the movement range as the viewpoint position. In other words, the renderer 220 may fix the viewpoint position at the position (boundary position) at the time when the viewpoint position is about to move outside the movement range.

Second, when the user's viewpoint position moves outside the movement range, the renderer 220 may notify the user that movement of the viewpoint position is restricted. For example, the renderer 220 may display a message such as "You cannot move beyond this point."

(Action and Effects)
In the embodiment, media processing device 200 generates specific content based on viewpoint information and transmits the specific content to user terminal 300. With this configuration, there is no need for user terminal 300 to generate specific content including second content with viewpoint freedom, and user terminal 300 can present the specific content by simply providing viewpoint information to media processing device 200. Therefore, although a delay occurs between media processing device 200 and user terminal 300, the processing load on user terminal 300 can be reduced.

In operation example 1, the media processing device 200 transmits the viewpoint information used in generating the specific content to the user terminal 300 in association with the sequence number added to the specific content. With this configuration, the user terminal 300 can grasp the viewpoint information and sequence number used in generating the specific content, and therefore can appropriately display the specific content even in cases where the viewpoint information used in generating the specific content by the media processing device 200 differs from the viewpoint information used in displaying the specific content on the user terminal 300.

In operation example 2, the media processing device 200 receives a content configuration and specific viewpoint information (recommended viewport information) from the transmitting device 100. With this configuration, the media processing device 200 can generate specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed.

In operation example 2, the media processing device 200 generates specific content based on specific viewpoint information in response to a reset signal, even when the user terminal is the first user terminal 400. With this configuration, even in a case where the viewpoint position and line of sight direction become unknown in the three-dimensional space constructed by the scene description in the first user terminal 400 (a case where the user becomes lost in the three-dimensional space), the reset signal can return to specific content based on specific viewpoint information.

In operation example 3, the media processing device 200 receives, from the transmitting device 100, quality information about each of two or more streams whose quality varies depending on the orientation of the 3D object based on the viewpoint information, as quality information about a stream related to a 3D object included in specific content. With this configuration, based on the new knowledge that the quality of each face constituting a bounding box for a 3D object does not have to be uniform, it is possible to appropriately display a 3D object while suppressing transmission traffic.

In operation example 4, the media processing device 200 receives importance information for each of two or more objects included in the specific content from the transmitting device 100. With this configuration, by introducing a mechanism for setting the importance of each object, such as a 360° video image or a 3D object, it is possible to appropriately display each object included in the specific content while suppressing transmission traffic.

In operation example 5, the media processing device 200 receives from the transmission device 100 an information element that defines the range of movement of the user's viewpoint position in the three-dimensional space configured by the specific content. With this configuration, it is possible to appropriately display the specific content, including content with freedom of viewpoint, without causing a breakdown in the specific content displayed on the user terminal 300.

[Change Example 1]
Modification 1 of the embodiment will be described below. Differences from the embodiment will be mainly described below.

In the first modification, a method for synchronizing a first content with a second content when a specific content includes both the first content and the second content is described.

Note that in the following, synchronization means that the presentation times of the first content (e.g., MPU) and the second content (file) are properly aligned. Therefore, synchronization may include the presentation times of the 2D video and the 3D object being aligned, and may also include the presentation times of the audio and the 3D object being aligned. Similarly, synchronization may include the presentation times of the 2D video and the 360° video being aligned, and may also include the presentation times of the audio and the 360° video being aligned.

In the first method, a case will be described in which the media processing device 200 synchronizes the first content and the second content based on the first control information (MMT-SI). Using the MMT-SI as an entry point, the media processing device 200 checks whether or not a scene description (second content) is present, and if a scene description is present, it uses the MPU timestamp descriptor to identify the presentation time of specific content including the first content and the second content.

Specifically, as shown in FIG. 19, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 19), so that the 2D video and audio can be synchronized.

On the other hand, the presentation time of the first frame included in the scene description is identified by referencing the MPU timestamp descriptor included in MMT-SI. The presentation times of the second and subsequent frames included in the scene description can be identified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is identified by adding 1/30 x n to the time identified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

In the first method, a case is exemplified in which the scene description does not include the presentation time of the first frame included in the scene description, but the scene description may include the presentation time of the first frame included in the scene description.

In the second method, a case will be described in which the media processing device 200 synchronizes the first content and the second content based on the second control information (scene description). The media processing device 200 uses the scene description as an entry point to check whether or not MMT-SI (first content) is present, and if MMT-SI is present, identifies the presentation time of specific content, including the first content and the second content, based on the presentation time included in the scene description.

In such a case, the scene description includes absolute time information indicating the presentation time of the second content. The absolute time information may be the presentation time of the first frame included in the scene description.

For example, the absolute time information may be generated using UTC as the reference time. The reference time may be the TAI or may be the time provided by a GPS. The reference time may be the time provided by an NTP server or may be the time provided by a PTP server. Furthermore, the absolute time information may be generated based on the same reference time as the MPU timestamp descriptor.

Furthermore, the scene description includes reference information for identifying the first content. The reference information may be information for identifying an MPU that constitutes the first content. In other words, the reference information is information for treating the first content (MPU) as an object included in the scene description.

Specifically, as shown in FIG. 20, the presentation time of the first frame included in the scene description is specified by the absolute time information included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1/30×n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

On the other hand, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 20), so that the 2D video and audio can be synchronized. Here, because the above-mentioned reference information is included in the scene description, the media processing device 200 can check whether or not there is a first content that should be presented together with the second content, based on the reference information included in the scene description.

In the second method, synchronization between 2D video and audio is achieved based on the MPU timestamp descriptor included in MMT-SI, but in modification example 1, synchronization between 2D video and audio may also be achieved based on information elements included in the scene description (absolute time information and reference information). In such a case, at least the MPU timestamp descriptor included in MMT-SI may be omitted. Furthermore, MMT-SI itself may be omitted.

When the reference time of the MPU timestamp descriptor included in the MMT-SI (hereinafter, the first reference time) differs from the reference time of the absolute time information included in the scene description (the second reference time), at least one of the first control information (MMT-SI) and the second control information (scene description) may include conversion information between the first reference time and the second reference time. For example, the MMT-SI may include an MPU timestamp descriptor expressed in the second reference time (e.g., a reference time other than UTC) in addition to the MPU timestamp descriptor expressed in the first reference time (e.g., UTC). The scene description may include absolute time information expressed in the first reference time (e.g., UTC) in addition to the absolute time information expressed in the second reference time (e.g., a reference time other than UTC).

The MPU timestamp descriptor included in the MMT-SI may be referred to as the first absolute time information, and the absolute time information included in the scene description may be referred to as the second absolute time information.

[Change Example 2]
In the following, a second modification of the embodiment will be described. In the following, differences from the embodiment will be mainly described. Specifically, in the second modification, a scene description editing device used for editing a scene description that defines a specific content will be mainly described.

(Media processing device and scene description editing device)
The following describes a media processing device and a scene description editing device according to Modification 2. Figure 21 is a block diagram showing a media processing device 200 and a scene description editing device 600 according to Modification 2.

First, the media processing device 200 has a reception unit 210, a renderer 220, and an encoding processing unit 230.

The reception unit 210 receives the viewpoint information. In modification example 2, the reception unit 210 may receive the viewpoint information from the scene description editing device 600.

The renderer 220 generates specific content that includes at least the second content based on the viewpoint information. In the following, an example is given of the specific content including the first content in addition to the second content.

First, the renderer 220 generates a first content including 2D video and audio as part of a specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.

In the above-described embodiment, the media processing device 200 receives the first control information (MMT-SI), 2D video, and audio from the transmitting device 100, but in modification 2, the media processing device 200 receives the first control information (MMT-SI), 2D video, and audio from the scene description editing device 600.

Secondly, the renderer 220 generates second content including the 360° video and the 3D object as part of the specific content based on the second control information (scene description). The viewpoint information is used in generating the second content.

In the above-described embodiment, the media processing device 200 receives the second control information (scene description), the 360° video, and the 3D object from the transmission device 100, but in modification example 2, the media processing device 200 receives the second control information (scene description), the 360° video, and the 3D object from the scene description editing device 600.

The encoding processing unit 230 encodes the specific content generated by the renderer 220. In a second modification, the encoding processing unit 230 may transmit the specific content to the scene description editing device 600.

Second, the scene description editing device 600 has a detection unit 610, a decoding processing unit 620, a renderer 630, an editing unit 640, and a database 650.

The detection unit 610 detects the user's viewpoint position and line of sight direction, similar to the detection unit 310 described above. The detection unit 610 may include an acceleration sensor or a GPS sensor. The detection unit 610 may include a user I/F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) that is manually input by the user. The detection unit 610 may transmit viewpoint information (viewpoint position and line of sight direction) to the media processing device 200. The detection unit 610 may output viewpoint information (viewport) to the renderer 630.

The decoding processing unit 620 decodes the specific content received from the media processing device 200, similar to the decoding processing unit 320 described above. The decoding processing unit 620 may decode the presentation time received from the media processing device 200. The decoding processing unit 620 may output the specific content to the renderer 630, or may output the presentation time to the renderer 630.

Similar to the renderer 330 described above, the renderer 630 outputs the specific content decoded by the decoding processing unit 620. The renderer 630 may output the specific content based on the presentation time decoded by the decoding processing unit 620. For example, the renderer 630 may output the video content included in the specific content to a display and output the audio content included in the specific content to a speaker.

In the second modification, the specific content output from the renderer 630 is used for editing scene descriptions, etc., and may therefore be considered a preview of the specific content.

The editing unit 640 is a component used to edit a scene description that defines the configuration of content (second content) that has freedom of viewpoint. The editing unit 640 may transmit the scene description to the media processing device 200. When the scene description is updated, the editing unit 640 may transmit the updated scene description to the media processing device 200. Updating the scene description may be interpreted as editing the scene description. The editing unit 640 may include a user I/F (e.g., a touch sensor, a keyboard, a mouse, a controller, etc.) for editing the scene description.

As described above, the scene description is generated for each file, and includes information for each frame that identifies the 360° video and 3D objects. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, an information element (scale_object) indicating the size of the 3D object in the frame, etc.

The database 650 stores media such as 3D objects, 2D video, audio, and 360° video. The database 650 may transmit required media files to the media processing device 200 in response to a transmission request or a read request from the media processing device 200.

(UI of scene description editor)
The following describes an example of a UI (User Interface) of the scene description editing device according to Modification 2. Fig. 22 is a diagram showing an example of the UI of the scene description editing device 600 according to Modification 2. The UI of the scene description editing device 600 may be considered as an editing screen for editing the scene description.

As shown in FIG. 22, the editing interface (UI) may include a media list, a timeline, a preview, a scene description, and the like.

The media list is an area that displays a list of media such as 3D objects, 2D video, audio, and 360° video specified by the creator of a particular content (hereinafter simply referred to as the creator).

The timeline is an area where the creator selects elements that make up a scene from a media list, and the selected elements are arranged on the timeline. The timeline includes an indicator (such as a line) that indicates the time selected by the creator on the timeline, and the scene at the time indicated by the indicator may be displayed in the preview as a 2D image of the specific content.

The preview is an area in which specific content is displayed according to viewpoint information (viewpoint position and viewpoint direction) specified by the creator's operation. The viewpoint information may be detected by the detection unit 610 described above. The creator may edit the scene description while checking the specific content displayed in the preview. The preview may be displayed by a head-mounted display used by the creator. In addition, two or more previews may be provided, such as displaying the same scene at the same time from different viewpoints, such as looking downward from the sky or looking east to west. By displaying the same scene from multiple different viewpoints, it is possible to edit the scene description while easily checking the arrangement of objects in three-dimensional space.

The scene description is an area where the scene description edited by the creator is displayed. The scene description may include specific viewpoint information (recommended viewport information) described in operation example 2 (see, for example, Figure 7). The scene description may include quality information and placement information described in operation example 3 (see, for example, Figure 10). The scene description may include importance information described in operation example 4 (see, for example, Figure 14). The scene description may include information elements that define the movement range of the user's viewpoint position described in operation example 5 (see, for example, Figures 17 and 18).

(Example of operation)
An editing operation using the scene description editing device 600 will now be described.

First, when the creator starts editing the scene description, the scene description editing device 600 may notify the media processing device 200 of the start of editing the scene description by RTSP SETUP. The RTSP SETUP may include the IP address of the scene description editing device 600, a listening port number, a request to operate in edit mode, and the like.

Secondly, the viewpoint information (viewpoint position and viewpoint direction) can be changed by the creator's operation. When the viewpoint information is changed by the creator's operation, the changed viewpoint information is transmitted from the scene description editing device 600 to the media processing device 200. This mechanism is similar to the mechanism for transmitting viewpoint information from the user terminal 300 to the media processing device 200.

Third, parameters such as the position, size, and orientation of elements such as 3D objects can be changed by the creator's operation (editing). When parameters are changed by the creator's operation, the changed parameters are reflected in the scene description. The changed scene description is transmitted from the scene description editing device 600 to the media processing device 200 and rendered by the renderer 220 of the media processing device 200, and the rendered specific content is transmitted from the media processing device 200 to the scene description editing device 600. In other words, the specific content displayed in the preview is updated in response to the change in parameters. The mechanism by which the specific content based on the scene description is transmitted from the media processing device 200 to the scene description editing device 600 is similar to the mechanism by which the specific content is transmitted from the media processing device 200 to the user terminal 300.

Fourth, the creator may select icon 710 ("recommended view" in FIG. 22) included in the editing screen (UI) to display specific content in the preview that corresponds to the specific viewpoint information (recommended viewport information) described in operation example 2.

Fifth, viewpoint information corresponding to specific content displayed in the preview may be recorded as specific viewpoint information (recommended viewport information) described in operation example 2 by the creator selecting icon 720 ("recommended view record" in FIG. 22) included in the editing screen (UI). The specific viewpoint information (recommended viewport information) recorded by icon 720 may be reflected in the scene description. For example, viewpoint information corresponding to specific content displayed in the preview may be operated by a head mounted display, and the viewpoint information operated by the head mounted display may be recorded as specific viewpoint information (recommended viewport information) by selecting icon 720. In other words, the specific viewpoint information is generated in response to the operation of the head mounted display used by the creator of the specific content.

Sixth, the information element (for example, "Viewing_space" shown in FIG. 18) that defines the movement range of the user's viewpoint position described in operation example 5 may be specified by the creator's operation (such as mouse operation). The information element ("Viewing_space") that defines the specified movement range may be reflected in the scene description. It may be possible to switch whether or not to display the information element ("Viewing_space") that defines the movement range (display or non-display).

In the second modification, the scene description editing device 600 may be implemented as a separate entity from the media processing device 200, and may be connected to the media processing device 200 via a network. For example, the media processing device 200 may be implemented as Software as a Service (SaaS) on the cloud. However, the second modification is not limited to this. For example, the scene description editing device 600 may be implemented as a single device (computer, hardware) integrated with the media processing device 200. In such a case, the encoding processing unit 230 and the decoding processing unit 620 described above may be omitted. In addition, transmission may be read as output, and reception may be read as acquisition.

(Action and Effects)
In Modification 2, the scene description editing device 600 acquires the specific content generated by the media processing device 200 based on the scene description and the viewpoint information. With this configuration, assuming a case in which the specific content is generated (rendered) by the media processing device 200 and then the generated specific content is transmitted from the media processing device 200 to the user terminal 300, the scene description editing device 600 can acquire the specific content from the media processing device 200 in a manner similar to that of the user terminal 300. Therefore, it is possible to appropriately edit the scene description while checking the specific content that is expected to be displayed on the user terminal 300.

[Other embodiments]
Although the present invention has been described by the above disclosure, the description and drawings forming a part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art.

In the above disclosure, a case in which the specific content includes both the first content and the second content has been exemplified, but the above disclosure is not limited to this. The specific content may include at least the second content.

Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents specified in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.

In the above disclosure, an MPU timestamp descriptor is exemplified as the first absolute time information included in MMT-SI. However, the above disclosure is not limited to this. The first absolute time information included in MMT-SI may be an MPU extended timestamp descriptor.

Although not specifically mentioned in the above disclosure, the media processing device 200 may request a portion of the second content from the transmitting device 100 as necessary. This configuration can save bandwidth associated with the transmission of the second content and suppress an increase in the processing load on the media processing device 200.

In the above disclosure, MMTP is exemplified as the transmission method of the first content. However, the above disclosure is not limited to this. The transmission method of the first content may be a method conforming to ISO/IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the first control information may be MPD (Media Presentation Description). That is, in the above disclosure, MMT-SI may be read as MPD.

Although not specifically mentioned in the above disclosure, "obtain" may be read as "receive."

Although not particularly limited, the operation example 2 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a content configuration having viewpoint freedom, and the transmitting unit transmits specific viewpoint information used to generate specific content including at least the content. The receiving device includes a receiving unit that receives a content configuration having viewpoint freedom, and the receiving unit receives the specific viewpoint information used to generate specific content including at least the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

Although not particularly limited, the operation example 3 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits quality information of a stream related to a three-dimensional object included in specific content that includes at least the content, and the quality information includes quality information related to each of two or more streams whose quality differs depending on the orientation of the three-dimensional object. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives quality information of a stream related to a three-dimensional object included in specific content that includes at least the content, and the quality information includes quality information related to each of two or more streams whose quality differs depending on the orientation of the three-dimensional object. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

Although not particularly limited, operation example 4 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a content configuration having freedom of viewpoint, and the transmitting unit transmits importance information regarding each of two or more objects included in specific content that at least includes the content. The receiving device includes a receiving unit that receives a content configuration having freedom of viewpoint, and the receiving unit receives importance information regarding each of two or more objects included in specific content that at least includes the content. In such a case, the receiving device may be a media processing device 200 or a user terminal 300.

Although not particularly limited, the operation example 4 may be expressed as follows. The transmitting device 100 includes a transmitting unit that transmits a configuration of content having freedom of viewpoint, and the transmitting unit transmits information elements that define a movement range of a user's viewpoint position in a three-dimensional space configured by specific content that includes at least the content. The receiving device includes a receiving unit that receives a configuration of content having freedom of viewpoint, and the receiving unit receives information elements that define a movement range of a user's viewpoint position in a three-dimensional space configured by specific content that includes at least the content. In such a case, the receiving device may be the media processing device 200 or the user terminal 300.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the transmitting device 100, the media processing device 200, and the user terminal 300. The program may also be recorded on a computer-readable medium. Using a computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or DVD-ROM.

Alternatively, a chip may be provided that is configured with a memory that stores programs for executing the processes performed by the transmitting device 100, the media processing device 200, and the user terminal 300, and a processor that executes the programs stored in the memory.

(Additional Note)
The above disclosure may be expressed as follows:

The first feature is a scene description editing device that includes an output unit that outputs a scene description and viewpoint information that define the configuration of content with viewpoint freedom to a media processing device, and an acquisition unit that acquires specific content that includes at least the content and is generated by the media processing device based on the scene description and the viewpoint information.

The second feature is a scene description editing device according to the first feature, in which the specific content generated by the media processing device is the specific content that is transmitted to a user terminal used by a viewer of the specific content.

The third feature is the scene description editing device according to the first or second feature, wherein, when the scene description is updated, the output unit outputs the updated scene description to the media processing device.

The fourth feature is a scene description editing device in at least one of the first to third features, in which the output unit outputs specific viewpoint information applied to the specific content to the media processing device.

The fifth feature is a scene description editing device in at least one of the first to fourth features, in which the specific viewpoint information is generated in response to an operation of a head-mounted display used by the creator of the specific content.

The sixth feature is a scene description editing device in at least one of the first to fifth features, in which the output unit outputs to the media processing device information elements that define a range of movement of a user's viewpoint position in a three-dimensional space formed by the specific content.

10...transmission system, 100...transmission device, 200...media processing device, 210...reception unit, 220...renderer, 230...encoding processing unit, 260...selection unit, 270...selection unit, 300...user terminal, 310...detection unit, 320...decoding processing unit, 330...renderer, 400...first user terminal, 500...second user terminal, 600...scene description editing device, 610...detection unit, 620...decoding processing unit, 630...renderer, 640...editing unit, 650...database

The present invention relates to a scene description editing device and a program .

Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a scene description editing device and program that enables appropriate editing of scene descriptions that define specific content when the rendering function of a media processing device is used in a user terminal.

According to the present invention, it is possible to provide a scene description editing device and program that enable appropriate editing of a scene description that defines specific content when the rendering function of a media processing device is used in a user terminal.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the transmitting device 100, the media processing device 200 , the user terminal 300 , and the scene description editing device 600. The program may be recorded on a computer-readable medium. The computer-readable medium can be used to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

Alternatively, a chip may be provided that is configured by a memory that stores programs for executing the processes performed by the transmitting device 100, the media processing device 200 , the user terminal 300 , and the scene description editing device 600, and a processor that executes the programs stored in the memory.

Claims (6)

an output unit that outputs a scene description and viewpoint information that define a configuration of content having viewpoint freedom to a media processing device;
an acquisition unit that acquires specific content including at least the content, the specific content being generated by the media processing device based on the scene description and the viewpoint information. The scene description editing device according to claim 1, wherein the specific content generated by the media processing device is the specific content that is transmitted to a user terminal used by a viewer of the specific content. The scene description editing device according to claim 1, wherein the output unit outputs the updated scene description to the media processing device when the scene description is updated. The scene description editing device according to claim 1, wherein the output unit outputs specific viewpoint information applied to the specific content to the media processing device. The scene description editing device according to claim 4, wherein the specific viewpoint information is generated in response to an operation of a head-mounted display used by the creator of the specific content. The scene description editing device according to claim 1, wherein the output unit outputs to the media processing device information elements that define a range of movement of a user's viewpoint position in a three-dimensional space formed by the specific content.

Images