JP2018125621A - Client, program and method for controlling reception of segment of data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a client capable of appropriately performing request for a more suitable segment file depending on a communication situation of a communication network, in receiving transmission data divided into segments.SOLUTION: The client receives, from a server, transmission data divided into segments, for each of which a plurality of types of segment files are prepared. In receiving a single segment, the client transmits, to the server, a request for acquiring one or a part of a plurality of types of segment files prepared for the single segment.The client then sequentially transmits requests for acquiring at least one of the remaining segment files that have not yet been requested. After receiving the requested files, the client executes processing on the received segment files for the single segment. Preferably, a request for acquiring a segment file having higher set priority is transmitted more preferentially.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a reception control technique in a client that acquires a large amount of data from a server.

  Free viewpoint video is attracting attention as one of the media that can provide a high level of realism in the next generation. A free viewpoint video is a video that enables interactive viewing in which a user arbitrarily specifies a viewpoint with respect to an object and provides a video from a specified viewpoint as needed.

As a typical example of this free viewpoint video composition method, an object image for each viewpoint is placed on a three-dimensional model, and the appearance from an arbitrary virtual viewpoint specified by the user is calculated using these placement images. By doing so, there is a method of performing drawing. Specifically, the composition processing in this method is performed on the client side that has acquired the content including the following (a) to (c) distributed from the server.
(A) Video stream at each viewpoint (b) Object information existing at each viewpoint (mask image, information on three-dimensional position / size)
(C) Camera calibration information for each viewpoint

  As described above, in the free viewpoint video composition processing, it is normal that the server side needs to prepare each content for the number of viewpoints (for multiple viewpoints) in advance. At this time, in particular, the video stream of (a) above has a considerably high bit rate per viewpoint due to higher resolution, and the amount of data becomes too large considering distribution via the Internet. Has occurred.

  To deal with this problem, for example, in Patent Document 1, while distributing video streams for all viewpoints, reducing the bit rate of videos other than the viewpoint currently designated by the user, thereby reducing the overall rate as much as possible. Techniques to make it have been proposed.

  On the other hand, as a free viewpoint video distribution method, for example, when video content created in advance is distributed, a VoD (Video on Demand) type distribution is generally used. Specifically, protocols such as DASH (Dynamic Adaptive Streaming over Hyper transfer protocol) and HLS (HTTP Live Streaming) are widely used. In such a system, the video stream is divided into segment files having a playback time length of about several to 10 seconds in advance, and the client side receives and starts playback in units of segments, thereby receiving the video stream. Playing pseudo streaming is realized.

  Furthermore, by preparing video streams of different qualities (bit rates) and preparing segment files having different bit rates for each segment, the bit rate can be switched in units of segments. As a result, the quality of the content requested by the client can be changed according to the communication status of the communication network.

JP 2013-183209 A

  However, in the conventional technology including the technology described in Patent Document 1, depending on the communication status of the communication network used for distribution, a segment file reception delay occurs on the client side. It has become a problem that it may stop or suffer some kind of trouble.

  In fact, in the technique described in Patent Document 1, the client side determines the bit rate distribution in the requested segment file for each viewpoint based on the probability distribution that maximizes the point of sight, and then the server receives the client from the client. Upon receipt of the request, the segment file of each viewpoint having the determined bit rate is multiplexed and distributed.

  For this reason, the timing for determining the bit rate distribution in the segment file for each viewpoint and requesting the segment file for each viewpoint is the same for all viewpoints. Therefore, for example, even when the throughput of the communication network suddenly decreases after a segment file request, it is impossible to respond such that the bit rate of the segment file to be delivered is lowered later. .

  That is, in the prior art, specifically, reproduction (decoding) cannot be started and the distribution of bit rates in the segment files for each viewpoint cannot be changed until the reception of the segment files for all viewpoints is completed. As a result, for example, the bit rate for viewpoints other than the viewpoint currently selected by the user (hereinafter referred to as a gazing point) is adaptively lowered, and the reception of segment files for all viewpoints is made in time for the next playback start. Control is impossible. As a result, in a situation where the throughput frequently fluctuates, it is not possible to cope with dynamic distribution control for each viewpoint, and there is a problem that playback is likely to stop.

  Furthermore, since the bit rate allocation for each viewpoint is determined independently of the allocation at the time of the previous request, as a result, the bit rate in the segment file of the gazing point is also dragged when throughput decreases. There was also a problem of a drop.

  Incidentally, the problem of processing stop / interruption on the client side due to such reception delay is not limited to the case of the multi-view video stream exemplified above. For example, if the client is a large-capacity transmission data divided into a plurality of segment files and requests data to a server capable of transmitting transmission data in which a plurality of types of segment files are prepared for each segment A similar problem occurs in reality.

  Therefore, the present invention provides a client, a client control program, and a client control method capable of appropriately requesting a segment file in accordance with the communication status of a communication network when receiving transmission data divided into segments. For the purpose.

According to the present invention, the transmission data divided into a plurality of segments, a client for a server capable of transmitting transmission data in which a plurality of types of segment files are prepared for each segment,
When one segment is received, a request for obtaining one or a part of the plurality of types of segment files prepared as one segment is transmitted to the server, and then requests are sequentially made. Data request means for causing a request to acquire at least one of the remaining segment files not to be transmitted to the server;
There is provided a client having data processing means for executing processing of a segment file for one received segment.

  The client data request means according to the present invention sets priorities for the plurality of types of segment files to be acquired based on received instructions or predetermined matters, and acquires the segment file with the highest priority. It is also preferable that the request to be transmitted is given priority.

Here, as a specific example of the embodiment using the above-described priority, the transmission data is video and / or audio stream data, and the plurality of types of segment files are mutually different in viewpoint and / or listening point. Different types of video and / or audio segment files,
It is also preferable that the data requesting unit sets the highest priority for the video and / or audio segment file related to the gaze point and / or the gaze point.

As another embodiment of the client according to the present invention, a plurality of segment files having different data amounts are prepared for each of the plurality of types of segment files.
The data requesting means may determine which segment file to select for each segment file to be acquired when receiving the next segment, based on the received instruction or predetermined matters. preferable.

Furthermore, as yet another embodiment of the client according to the present invention, a plurality of segment files having different data amounts are prepared for each of the plurality of types of segment files.
When the data request means receives one segment, the data request means determines the stage of the segment file to be acquired next or thereafter based on the reception status of the segment file received as a result of one request. It is also preferable.

  Further, in the embodiment for determining the stage of the segment file, the data requesting unit is configured to request the subsequent acquisition of the plurality of types of video and / or audio segment files based on the degree of reception of the segment file at the certain stage. It is also preferable to determine a bit rate distribution in which a bit rate to be selected is assigned to each of the above.

Here, as one specific example of the embodiment for determining the stage of the segment file, the transmission data is video and / or audio stream data, and the plurality of types of segment files are mutually viewpoints and / or listening points. A plurality of types of video and / or audio segment files having different bit rates, the segment files having different bit rates from each other,
The data requesting means determines the bit rate of the segment file to be acquired next or thereafter based on the throughput calculated when receiving the segment file of a certain bit rate received as a result of one request. Is also preferable.

Furthermore, as yet another embodiment of the client according to the present invention, the client determines whether reception of one segment is completed at least by the start of reception of the next segment to be received next. A reception delay determining means for determining;
In receiving the one segment, the data requesting unit preferably determines the number of segment files to be acquired in each subsequent request based on the determination result of the reception delay determining unit.

  In the embodiment for performing the reception delay determination, when the data requesting unit requests acquisition of a plurality of segment files in one request to be performed thereafter, the data requesting unit includes a push transmission instruction from the server in the one request. Inclusion is also preferred.

According to the present invention, a computer mounted on a client for a server capable of transmitting transmission data divided into a plurality of segments, in which a plurality of types of segment files are prepared for each segment. A client control program,
Because
When one segment is received, a request for obtaining one or a part of the plurality of types of segment files prepared as one segment is transmitted to the server, and then requests are sequentially made. Data request means for causing a request to acquire at least one of the remaining segment files not to be transmitted to the server;
A client control program is provided that causes a computer to function as data processing means for executing processing of a received segment file for one segment.

According to the present invention, a client in a computer mounted on a client for a server capable of transmitting transmission data that is divided into a plurality of segments and in which a plurality of types of segment files are prepared for each segment. A control method,
When one segment is received, a request for obtaining one or a part of the plurality of types of segment files prepared as one segment is transmitted to the server, and then requests are sequentially made. Sending a request to the server to obtain at least one of the remaining remaining segment files;
A client control method is provided that includes processing a received segment file for one segment.

  According to the client, the client control program, and the client control method of the present invention, when receiving transmission data divided into segments, a more suitable segment file request can be implemented as appropriate according to the communication status of the communication network. .

It is a functional block diagram which shows the function structure in one Embodiment of the client by this invention. It is a schematic diagram which shows one Embodiment of the multiview video stream data prepared in a delivery server. It is a sequence diagram which shows the outline of communication with the HTTP client and HTTP server by DASH. It is the schematic diagram which showed the delivery form of the conventional segment file schematically. It is a schematic diagram for demonstrating the positional relationship of the viewpoint selected by the user in a smart phone (client). It is the schematic diagram which showed roughly one Embodiment of the delivery control of the segment file by this invention. It is the schematic diagram which showed schematically embodiment of segment file delivery control corresponding to the reception delay by this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a functional block diagram showing a functional configuration in an embodiment of a client according to the present invention.

  A smartphone 1 according to the present embodiment illustrated in FIG. 1 is a client (terminal) that requests a distribution server 2 to transmit a large amount of transmission data via a communication network, and receives and processes the transmission data. .

  Specifically, in this embodiment, the transmission data stored in the distribution server 2 is “multi-view video stream data” and is divided into a plurality of segments, and a plurality of types of segment files are prepared for each segment. ing. In the present embodiment, the plurality of types of segment files are a plurality of video data having different viewpoints. The smartphone 1 acquires (receives) a plurality of segment files having different viewpoints (multi-viewpoints) and synthesizes them to generate a free viewpoint video.

  In the present embodiment, “multi-view video stream data” is transmitted / received based on the HTTP (HyperText Transfer Protocol) streaming method for multi-view video, and the protocol is specifically DASH (Dynamic Adaptive Streaming over Hyper transfer). protocol) is used. Also, the distribution server 2 as an HTTP server distributes “multi-view video stream data” as video content created in advance using a VoD (Video on Demand) type distribution method. Naturally, the distribution method and adoptable protocol of “multi-view video stream data” are not limited to those described above.

  As described above, the “multi-view video stream data” needs to be prepared in advance for the number of viewpoints (multi-viewpoints) for the video content divided into segments in the distribution server 2, and the video of each viewpoint is also prepared. The stream usually has a very high bit rate (quality) due to the higher resolution. Therefore, depending on the communication status of the communication network, considerable ingenuity and consideration are required for the distribution of “multi-view video stream data” in order to perform video playback processing on the smartphone 1 without delay.

Also in FIG. 1, the smartphone 1 is characterized by its prominent features:
(A1) When one segment is received from the distribution server 2, one of a plurality of types of segment files (a plurality of video data having different viewpoints in this embodiment) prepared as one segment Alternatively, a request for acquiring a part (HTTP Request in the present embodiment) is sent to the distribution server 2,
(A2) Next, sequentially send a request (HTTP Request) to acquire at least one of the remaining segment files not requested to the distribution server 2,
(B) Processing of the received segment file for one segment, in this embodiment, video playback processing is executed.

  As described above, when receiving one segment, the smartphone 1 does not acquire all the segment files of each viewpoint to be acquired by one request (HTTP Request), but one or a part of them is acquired. To get. Therefore, for example, when a change in the communication status of the communication network occurs after this request, it is possible to adjust the contents of the remaining segment file requests in response to this change. In other words, by applying the above configurations (A1) and (A2) in particular, a more suitable segment file request can be implemented as appropriate according to the communication status of the communication network.

  Furthermore, in the distribution server 2 of the present embodiment, a plurality of segment files having different bit rates (quality) are prepared for each of a plurality of types (multi-view) segment files having different viewpoints. Here, when the smartphone 1 receives one segment, the smartphone 1 receives a segment file of a certain bit rate received as a result of one request (HTTP Request), for example (for a communication network used). Based on the throughput, the bit rate of the next segment file to be acquired can be determined or requested, and the determined segment file can be requested.

  In this regard, in the prior art, the distribution of the bit rate in the segment file for each viewpoint cannot be changed until the reception of the segment files for all viewpoints is completed. As a result, for example, even in a situation where the throughput frequently fluctuates, it is not possible to cope with the dynamic distribution control for each segment file of each viewpoint, and there is a problem that playback is likely to stop.

  On the other hand, the smartphone 1 according to the present embodiment, even in a situation where the throughput frequently fluctuates, appropriately has a bit rate segment suitable for the communication situation until the completion of reception of the segment files for all viewpoints. It is possible to reselect and request a file to suppress playback stop.

  Incidentally, the problem of processing stop / interruption on the client side due to such reception delay is not limited to the case of the multi-view video stream exemplified above. For example, if the client is a large-capacity transmission data divided into a plurality of segments and is a client that makes a data request to a server capable of transmitting transmission data in which a plurality of types of segment files are prepared for each segment, A similar problem occurs in reality. According to the client of the present invention, it is possible to solve the problem of stop / interruption of processing related to such transmission data.

[Functional configuration of the device]
Similarly, according to the functional block diagram shown in FIG. 1, the smartphone 1 as the client of the present invention includes a communication interface unit 101, a touch panel display (TP / DP) 102, and a processor memory. Here, the processor memory realizes data request, reception, and processing functions, which are an embodiment of the client control method according to the present invention, by executing a program that causes the computer of the smartphone 1 to function.

  Further, the processor memory includes a data request unit 111 including a request viewpoint determination unit 111a, a request bit rate determination unit 111b, and a request timing determination unit 111c, a throughput calculation unit 112a, and a reception delay determination unit 112b as functional components. A communication management unit 112 including a viewpoint position detection unit 113a, a free viewpoint synthesis unit 113b, a media engine 113c, and a data processing unit 113 including a reception history recording unit 113d. Here, the flow of processing shown by connecting the functional components of the smartphone 1 in FIG. 1 with arrows is understood as an embodiment of the client control method according to the present invention.

The communication interface unit 101
(A) An HTTP Request (segment request) generated by the data request unit 111 described in detail later is transmitted to the distribution server 2 via a wireless communication network or a wired communication network such as the Internet. Also,
(B) As a response to the HTTP Request, the distribution server 2 receives the requested viewpoint and bit rate segment file.

The communication interface unit 101
(C) A processing result in the data processing unit 113 described in detail later, for example, synthesized free viewpoint video data, may be transmitted to an external information processing apparatus,
(D) The client control program (application) according to the present invention may be downloaded from an external server.

  FIG. 2 is a schematic diagram showing an embodiment of multi-view video stream data prepared in the distribution server 2.

  In FIG. 2, the distribution server 2 prepares multi-view video stream data to be stored and managed for distribution. First, for each of a plurality of set viewpoints (five viewpoints A to E in FIG. 2). In one multi-view video content including video data, for example, a plurality of stages of video data having different bit rates are prepared for each viewpoint using a known technique such as compression encoding. By the way, the bit rate is a numerical value (unit: bps (bits / sec), for example) that represents the quality of the target (video) data. For example, how much (video) information is packed in one second. It is the value shown.

  Next, for each of a plurality of viewpoints (five viewpoints A to E), video data of each bit rate is divided into a plurality of segments on the time axis. Here, a plurality of segments constituting one video data correspond to reproduction times 1, 2, 3,. One segment can have a reproduction time length of, for example, about several to 10 seconds. Furthermore, it is preferable that the playback time length of a segment constituting one video data is set to a constant value particularly in the case of VoD distribution.

  Through the preparation process described above, as shown in FIG. 2, the multi-view video stream data is prepared with a plurality of (four in FIG. 2) segment files having different bit rates for each playback time for each viewpoint. Stored and managed as a data set. In FIG. 2, the relative height (width in the height direction) of the rectangle indicating the segment file represents the level (step) of the bit rate. Further, for example, a segment file with a playback time “1” in video data of the viewpoint “A” is marked with a symbol “A1” to be distinguished from others.

  Incidentally, the transmission data that can be prepared by the distribution server 2 is not limited to the multi-view video stream data. For example, multi-listening point audio stream data generated by recording at a plurality of recording points can be handled. In this case, a plurality of audio data segments having a plurality of bit rates may be prepared. Further, the multi-view video stream data may include normal audio data that is not a multi-listening point, and a plurality of bit rate segments may be prepared for segments of the audio data. Further, multi-view video stream data including multi-point audio stream data can be used. In this case, a configuration as shown in FIG. 2 can be prepared for each of the video data and the audio data.

  In order to acquire multi-viewpoint video stream data prepared by the distribution server 2 as described above, the smartphone 1 (the communication interface unit 101) transmits and receives to and from the distribution server 2 based on DASH in this embodiment. To implement.

  FIG. 3 is a sequence diagram showing an outline of communication between an HTTP client and an HTTP server by DASH.

(S101) The HTTP client requests an MPD (Media Presentation Description) file in which video metadata is described to the HTTP server.
(S102) The HTTP server transmits the MPD file generated in advance to the requesting HTTP client.

(S103) The HTTP client makes a request (HTTP Request) for a segment file of playback time 1 having a preset bit rate to the HTTP server.
(S104) In response to the request in step S103, the HTTP server transmits the segment file of playback time 1 to the requesting HTTP client.

(S105) The HTTP client is using the size of the segment file received in step S104 and the time required to receive all (all viewpoints) segment files from the request time. Calculate network throughput. Next, the bit rate of the segment file that can be requested described in the MPD file received in step S102 is compared with the calculated throughput to determine the bit rate of the segment file to be requested next. To do.

  Here, as an example, when 5 Mbps, 2 Mbps, 1 Mbps, etc. can be requested as the bit rate, when the throughput is calculated as 2.5 Mbps, the bit rate of the next requested segment file exceeds the throughput. There can be no maximum value as 2Mbps.

(S106) The HTTP client passes the received segment file of playback time 1 to the media engine, and the media engine starts playback of the segment file (if the buffer is set to the length of one segment). .

(S107) The HTTP client makes a request (HTTP Request) for a segment file of playback time 2 having the bit rate determined in step S105 to the HTTP server.
(S108) In response to the request in step S107, the HTTP server transmits the segment file having the playback time 2 to the requesting HTTP client.

  Thereafter, the HTTP client and the HTTP server repeat the same procedure as S105 to S108, and sequentially request, transmit / receive, and reproduce the segment file after the reproduction time 3. Normally, a request (HTTP Request) by an HTTP client is periodically made every time a playback time length (for example, about several to 10 seconds) for one segment (medium) is elapsed.

  FIG. 4 is a schematic diagram schematically showing a distribution form of a conventional segment file.

  In FIG. 4, the HTTP client can use a bit that can be used with reference to the latest throughput (or according to a preset setting) at the timing of making an HTTP Request for requesting the segment files A1, B1, C1, D1, and E1 of playback time 1. The reference of the rate amount is determined, and further, the bit rate of the segment file of each viewpoint is determined based on the distribution of the bit rate distribution to each viewpoint set in advance.

  Next, the HTTP server that has received the request for the segment file of the determined bit rate multiplexes the segment files A1, B1, C1, D1, and E1 for all the corresponding viewpoints, and distributes them to the HTTP client as one file. . The HTTP client receives one file for all the viewpoints of the playback time 1 by the start time of the playback time 1, performs the playback process after the reception is completed, and performs segment files A2, B2 of the next playback time 2 , C2, D2, and E2 are requested.

  As described above, in the distribution of the conventional segment file, the acquisition of the segment file for all viewpoints is requested with one HTTP request. As a result, until the reception of the segment files for all viewpoints is substantially completed, It was impossible to start playback of the segment file without waiting for another reception, or to change the bit rate distribution of each viewpoint during the download.

  Here, a conventional bit rate determination process at the time of a segment request, specifically, a bit rate distribution method for each viewpoint segment file will be described. First, let b (t) be the total bit rate of all viewpoint segment files at time t. This b (t) is determined by selecting the highest bit rate that does not exceed the latest throughput from a plurality of bit rate candidates (described in the MPD file).

When the total bit rate b (t) is determined, the bit rate is then allocated to each viewpoint based on a preset selection probability distribution p (x _i | x _i-1 ) between the viewpoints. As this probability distribution p (x _i | x _i−1 ), for example, a normal distribution having a maximum value can be obtained when x _i = x _i−1 . Using this probability distribution, perspective x _i bit rates b _i (t) is at time t, the viewpoint previously selected as x _i-1, the following equation _{(1) b i (t)} = p (x _i | x _i-1 ) * b (t)
Can be determined by. Note that the latest throughput is calculated from the total size of segment files for all viewpoints in the previous playback time and the total time required for downloading.

  In this way, a segment file with the highest possible quality is requested by allocating the total bit rate b (t) determined according to the communication status of the network to each viewpoint to be requested.

Next, an embodiment of segment file distribution control according to the present invention will be described. The data request unit 111 (FIG. 1) of the smartphone 1 plays the role of this distribution control. When this data request unit 111 receives one segment,
(A) A request (HTTP Request) for acquiring one or a part of the multi-viewpoint (plural types) segment files prepared as one segment is transmitted to the distribution server 2,
(B) Next, a request (HTTP Request) for acquiring at least one of the remaining segment files not requested is sequentially transmitted to the distribution server 2.

  Incidentally, in this embodiment, the data request unit 111 includes a request viewpoint determination unit 111a that determines the viewpoint of the segment file in the request (HTTP Request), and a request bit rate determination that determines the bit rate of the segment file in the request (HTTP Request). Unit 111b and a request timing determination unit 111c that determines the timing of a request (HTTP Request). First, determination of the requested viewpoint (order) in the data request unit 111 (requested viewpoint determination unit 111a) will be described.

  FIG. 5 is a schematic diagram for explaining the positional relationship of viewpoints selected by the user in the smartphone 1.

  FIG. 5 shows the positional relationship between the viewpoint A, viewpoint B,. Each viewpoint occupies a predetermined range with respect to the viewpoint position represented by the direction angle θ (0 ° ≦ θ <360 °) as viewed from the subject. For example, in the example of FIG. 6, the viewpoint B is a viewpoint that virtually covers the range of 36 ° to 108 ° as the range of the viewpoint position. For example, the viewpoint position 60 ° belongs to this viewpoint B.

  The user can view the viewpoint position (or viewpoint) he / she wants to see by operating the viewpoint operation image displayed on the touch panel display 102 (FIG. 1), or by an operation (implicit) not related to such display. Is specified. The viewpoint designated here (related to the viewpoint position) is the gaze point. Here, the user may be able to change the designated viewpoint position at any time and continuously.

  The viewpoint position detection unit 113a (FIG. 1) constantly monitors a viewpoint instruction operation by the user, receives a gaze point instruction from the user, and sends the instruction content to the data request unit 111 (request viewpoint determination unit 111a) (FIG. 1). ). Here, the requested viewpoint determination unit 111a receives the input instruction content, determines a gazing point from the set viewpoints (for example, five viewpoints A to E), and further in the segment file of these viewpoints. The order (priority) to be requested is determined. As a result, it is possible to perform distribution control corresponding to viewpoint movement (viewpoint selection by the user) in the client.

  Incidentally, the requested viewpoint determination unit 111a (FIG. 1) sets priorities for not only received instructions but also multi-view (plural types) segment files to be acquired based on, for example, predetermined items, It is also preferable that a request for acquiring a segment file with a higher priority (a segment file with the highest priority point of interest) is transmitted with higher priority (highest priority).

  FIG. 6 is a schematic diagram schematically showing an embodiment of segment file distribution control according to the present invention.

  According to FIG. 6A, the smartphone 1 (data request unit 111) makes a total of five HTTP Requests when receiving one segment of the playback time 1. Specifically, an HTTP request is first made to request acquisition of the segment file C1 of the gazing point with the highest bit rate, and then the segment files B1 and D1 adjacent to the gazing point (closest to the gazing point) One HTTP Request is made to request one (B1 in FIG. 6A), and then the other HTTP Request to request D1 in FIG. 6A. The bit rates of these segment files B1 and D1 are determined to be the second highest value in this embodiment.

  Here, the segment files B1 and D1 are out of order with respect to the requests, but it is also preferable to request a viewpoint that is closer to the viewpoint as viewed from the viewpoint position (angle value) in consideration of the viewpoint position described above. For example, a viewpoint that has a viewpoint position (angle value) that is the center of the viewpoint position range to be covered closer to the viewpoint position (angle value) that is the center of the gazing point may be requested preferentially.

  After that, an HTTP Request is made to request one of the segment files A1 and E1 (A1 in FIG. 6A) that is farthest from the gazing point (second closest to the gazing point), and then the other (in FIG. 6A) E1) requesting HTTP request. The bit rates of these segment files A1 and E1 are determined to be the lowest (third highest) value in this embodiment. Incidentally, a distribution form is also possible in which these segment files A1 and E1 (not likely to be selected) are not required depending on the communication status or the like.

In the present embodiment, the data request unit 111 (required bit rate determination unit 111b) selects a segment file of any bit rate (stage) for each segment file to be acquired when receiving one segment. This can be determined in advance based on the setting items or instructions received from the user. In this case, it is also possible to employ a bit rate distribution process using the selection probability distribution p (x _i | x _i-1 ) at the time of the above-described conventional segment request.

  By performing such distribution processing, the data request unit 111 (required bit rate determination unit 111b) receives the next segment based on an instruction (or predetermined matter) received from the user. It is possible to determine a bit rate distribution in which a bit rate to be selected is assigned to each of all viewpoint segment files to be acquired. In this distribution, it is also preferable that the segment file for the gazing point is the segment file with the highest bit rate among the segment files for all viewpoints to be acquired.

  As described above, in the present embodiment, when receiving one segment, a segment file for each viewpoint is individually requested, and the gaze point designated (or preset) by the user is taken into consideration at that time. Then, the gaze point is requested first, and then the request is made in order from the gaze point (priority is given to a gaze point closer to the gaze point).

  Thus, for example, the bit rate of each viewpoint could not be changed in the middle of the download, but the smartphone 1 can change the bit rate as will be described next with reference to FIG. 6B. It becomes. Further, even before the reception of the segment files for all the viewpoints is completed, it is possible to start the reproduction of the gazing point segment file if necessary.

  Here, basically, the data request unit 111 of the smartphone 1 determines the requested order of segment files by viewpoint based on the time until playback, the point of sight, the throughput of the communication network being used, and the like. It is possible to determine the bit rate distribution to the viewpoint and the timing of making the request. Here, the throughput is calculated by the throughput calculation unit 112a (FIG. 1) of the communication management unit 112. The throughput calculation unit 112a can calculate the throughput at that time every time a request (HTTP Request) is made and one segment file is received, for example.

  Note that the timing of the request (HTTP Request) for each viewpoint is immediately after the previous requested segment file is received in the embodiment of FIG. 6A. It is not limited to. For example, in anticipation of this reception, a request (HTTP Request) for the next viewpoint can be made before the completion of this reception. Alternatively, a request (HTTP Request) may be made immediately after receiving one segment file, calculating the throughput based on this reception, and determining the next bit rate.

  Next, an embodiment of a dynamic change process of the bit rate by the smartphone 1 when the throughput of the communication network rapidly changes will be described.

  According to FIG. 6B, the throughput of the communication network in use is drastically decreasing at the time of requesting the segment file of playback time 1, specifically, immediately after the second HTTP Request transmission. In this case, the requested bit rate determining unit 111b (data requesting unit 111), based on the download result of the previously requested view segment file (B1 in FIG. 6B), the next requested view segment file (FIG. In 6 (B), the bit rate of D1) is determined.

  For example, the throughput is calculated from the data size of the segment file B1 and the time required for downloading, and this throughput is compared with the throughput (calculated in the same way) when the previous segment file C1 is downloaded, or It is determined whether or not it is lower than a predetermined level compared to a preset reference. When it is determined that the throughput has decreased by a predetermined value or more, the bit rate of the next requested segment file D1 can be dynamically changed to a smaller one. Further, the bit rate of the segment file A1 and the segment file E1 requested after that may be changed.

Here, a method of changing the distribution of the bit rate to each viewpoint will be described. In the embodiment of FIG. 6B, the total bit rate at all viewpoints to be subsequently requested at time t ′ when the download of the segment file B1 is completed is b ′ (t ′). Also, if the most recently measured throughput at time t ′ is TP, the remaining time until the playback start time of the segment file with playback time time 1 is Δt, and the playback time length of one segment file is l, then b ′ ( t ′) is the following conditional expression (2) b ′ (t ′) * l <TP * Δt
The maximum value satisfying the above can be determined. Alternatively, it may be determined by selecting from preset total bit rate candidate values that satisfy the above equation (2). Here, the latest throughput TP is calculated by a weighted average of throughput values calculated at the time of receiving each segment file received before time t ′ (for example, an average obtained by increasing the weight of the throughput calculated immediately before). It is also preferred that Further, for example, a value described in the MPD file can be adopted as the reproduction time length l in the above equation (2).

Next, when b ′ (t ′) is determined, the bit rate is allocated to each viewpoint based on the selection probability distribution p (x _i | x _i−1 ) between the viewpoints. The probability distribution p (x _i | x _i-1 ) can also be a normal distribution having a maximum value when, for example, x _i = x _i-1 . Using this probability distribution, the bit rate b _i ′ (t ′) of the viewpoint x _i at time t ′ is expressed by the following equation (3) b _i ′ (t ′) where x _i−1 is the viewpoint selected immediately before. ) = P (x _i | x _i-1 ) * b '(t')
Can be determined by.

Note that the selection probability distribution p (x _i | x _i−1 ) between the viewpoints may be updated to reflect the viewpoint selection situation up to time t ′. Specifically, for example, a history of viewpoint selection (determining the point of gaze) by the user is recorded, and a selection rate of each viewpoint in a predetermined time interval with time t ′ as an end point is calculated. The selection probability distribution p (x _i | x _i-1 ) can be updated so that the higher the viewpoint, the higher the probability value.

  In any case, the data request unit 111 (requested bit rate determining unit 111b) receives one segment (of a certain bit rate) as a result of one request (HTTP Request) when receiving one segment. Based on the reception condition, for example, the throughput calculated at the time of reception, the bit rate of the segment file to be acquired next or thereafter can be determined.

  Next, as another embodiment of the present invention, segment file distribution control corresponding to reception delay will be described.

Returning to FIG. 1, the reception delay determination unit 112b of the communication management unit 112 determines whether reception of one segment is completed at least by the start of reception of the next segment to be received next, that is, It is determined whether or not a reception delay occurs. Since the data request unit 111 receives one segment, the data request unit 111 determines the number of segment files that require acquisition in each subsequent request (HTTP Request) based on the determination result of the reception delay determination unit 112b. is there.

  FIG. 7 is a schematic diagram schematically showing an embodiment of segment file delivery control corresponding to reception delay according to the present invention.

  According to FIG. 7A, when the smartphone 1 (data request unit 111) receives one segment of the playback time 1, it first requests acquisition of the segment file C1 of the gazing point with the highest bit rate. HTTP request is made, and reception of this segment file C1 is completed. Next, the reception delay determination unit 112b (FIG. 1) refers to the reception delay determination result performed for the current time, thereby performing the following four HTTP request segment files B1, D1, and A1. Assuming that E1 and E1 are received individually, the reception completion is not in time for the next playback start.

  Therefore, as shown in FIG. 7B, the smartphone 1 (data request unit 111) requests a plurality of (two in the figure) segment files by one HTTP Request, and performs parallel reception of these files. . As a result, reception delay is avoided, and the video reproduction process is satisfactorily performed without interruption. Here, the parallel reception request can be implemented using, for example, an HTTP / 2 Server PUSH transmission function. More specifically, for example, in HTTP Request for requesting B1 (second request) shown in FIG. 7B, the segment file D1 to be pushed is specified in the GET header, and the distribution server 2 sets D1. An instruction for push transmission may be included.

Here, the reception delay determination by the reception delay determination unit 112b will be described. The reception delay determination unit 112b includes a round trip time (Round Trip Time) RTT and a throughput TP that have been measured recently (for example, by request / reception of the segment file C1), the number n of segment file groups to be received later, and The following formula using the total size S of segment files to be received (4) S / TP + (RTT / 2) * n> Δt
When is established, it is determined that a reception delay occurs. Here, Δt is the remaining time until the start time of playback time 2, which is the next playback time (the playback time of the segment file to be received this time). Further, n corresponds to (“number of HTTP Requests to be transmitted thereafter” −1). Eventually, the above expression (4) is a conditional expression in the case where the time required for completion of reception as viewed from the present time exceeds this Δt.

  Incidentally, the above reception delay occurs due to an increase in the total RTT accompanying an increase in the number of HTTP Requests. Therefore, the data request unit 111 that has received the determination result that a reception delay occurs requests a plurality of segment files with one HTTP Request, and avoids the expected reception delay by reducing the number of HTTP Requests to be transmitted. At this time, the data request unit 111 causes the reception delay determination unit 112b to determine whether there is a reception delay when the number of segment files requested in one HTTP Request is 2, 3,. It is also preferable to determine the minimum number satisfying (). In this way, by reducing the number of segment files requested in one HTTP Request as much as possible, more time is available for reviewing the bit rate for the viewpoint to be received during the reception of one segment. It is.

  Of course, if the reception delay is avoided, the number of segment files requested in each HTTP Request is not constant. After requesting two for the time being, for example, one HTTP Request according to the communication status at that time It is possible to request one segment file or three or more segment files. In addition, after requesting a segment file for a gazing point in the first HTTP Request, depending on the communication status, it is possible to request all the remaining segments all together and reliably receive all the viewpoints.

Next, the state of occurrence of reception delay in the distribution form shown in FIG. 7A and the distribution form in FIG. 7B will be specifically described. As a specific example, it is assumed that the time length of one segment is 10 seconds and RTT / 2 is 1 second. First, in the distribution form of FIG.
<Download (forecast) time><RTT / 2>
Segment file C1: 2 seconds 1 second Segment file B1: 1.5 seconds 1 second Segment file D1: 1.5 seconds 1 second Segment file A1: 0.5 seconds 1 second Segment file E1: 0.5 seconds 1 second The total time is 11 seconds. This is a value exceeding the time length of one segment (10 seconds), and it is determined that reception delay occurs as a result.

On the other hand, in the distribution form of FIG.
<Download (forecast) time><RTT / 2>
Segment file C1: 2 seconds 1 second File B1 and D1: 1.5 seconds + 1.5 seconds 1 second File A1 and E1: 0.5 seconds + 0.5 seconds 1 second. 9 seconds. This is a value that falls within the time length (10 seconds) of one segment, and as a result, it is determined that no reception delay occurs.

  In the embodiment of segment distribution control according to the present invention described with reference to FIGS. 6 and 7, the transmission data is multi-view video stream data, and the plurality of types of segment files are video segment files having different viewpoints. The multi-stage segment file is a video segment file having different bit rates, that is, image quality. However, as a change mode, the transmission data is multi-audience audio stream data, the plural types of segment files are made as audio segment files having different listening points, and the multi-stage segment files are mutually changed in bit rate, that is, sound quality. Different audio segment files may be used. Furthermore, stream data obtained by combining the multi-view video stream data and the multi-auditory audio stream data can be employed as the transmission data.

In any case, in the embodiment of the segment distribution control according to the present invention as described above with reference to FIGS. 6 and 7, each viewpoint and / or each listening in the multi-viewpoint and / or multi-listening point stream distribution. Point segment files are individually requested, and distribution control is performed dynamically as in the following (a) to (c).
(A) Request in order from the point of interest and / or the segment of the listening point, and further from the point of view and / or the segment of the listening point close to the point of interest.
(B) Based on the previously requested viewpoint and / or listening point download result, the bit rate allocation and the requested timing of the later requested viewpoint and / or listening point are dynamically determined.
(C) It is confirmed that there is no reception delay due to overhead in transmission of a plurality of HTTP Requests, and when reception delay is expected, parallel reception of segment files of a plurality of viewpoints and / or a plurality of listening points is performed by one HTTP Request.

  As a result, the bit rate of the segment file can be dynamically changed even in a situation where the throughput tends to fluctuate. As a result, the frequency at which playback stops or the playback image quality can be improved. it can.

  Returning to FIG. 1, the data processing unit 113 includes a free viewpoint synthesis unit 113b and a media engine 113c, and executes the segment file processing for each segment received from the distribution server 2 as described above. Specifically, the free viewpoint synthesis unit 113b generates a free viewpoint video by synthesizing the received segment files of each viewpoint, and outputs the generated viewpoint video to an application (application program) installed in the smartphone 1, for example. May use the free viewpoint video.

  The media engine 113c processes the received segment file of each viewpoint and displays the video of the gazing point on the touch panel display 102 or is selected when a user's viewpoint selection instruction is input from the viewpoint position detection unit 113a. It is also preferable to display the video of the selected viewpoint on the touch panel display 102. By the way, unlike the conventional case, if the smartphone 1 receives the segment file of the gazing point for the time being, the media engine 113c can also play back the gazing point without waiting for the completion of the reception of the segment file of the other viewpoint. It becomes.

  As described above in detail, according to the present invention, when receiving one segment, not all of the segment files for each viewpoint to be acquired are acquired by one request, but one of them is acquired. Or acquire a part. Therefore, a more suitable segment file can be requested as appropriate according to the communication status of the communication network to be used.

  Various changes, modifications, and omissions of the various embodiments of the present invention described above within the scope of the technical idea and the viewpoint of the present invention can be easily made by those skilled in the art. The above description is merely an example, and is not intended to be any limitation. The present invention is limited only by the claims and the equivalents thereof.

1 Smartphone (client)
101 Communication interface 102 Touch panel display (TP / DP)
111 Data Request Unit 111a Request Viewpoint Determination Unit 111b Request Bit Rate Determination Unit 111c Request Timing Determination Unit 112 Communication Management Unit 112a Throughput Calculation Unit 112b Reception Delay Determination Unit 113 Data Processing Unit 113a Viewpoint Position Detection Unit 113b Free Viewpoint Synthesis Unit 113c Media Engine 113d Reception history recording unit 2 Distribution server

Claims (11)

A transmission data divided into a plurality of segments, a client for a server capable of transmitting transmission data in which a plurality of types of segment files are prepared for each segment;
In receiving one segment, a request for obtaining one or a part of the plurality of types of segment files prepared as the one segment is transmitted to the server, and then sequentially requested. Data request means for sending a request to acquire at least one of the remaining segment files to the server;
And a data processing means for executing processing of the received segment file for one segment.   The data requesting means sets a priority for the plurality of types of segment files to be acquired based on the received instruction or predetermined matters, and further requests to acquire a segment file with a higher priority. The client according to claim 1, wherein the client is preferentially transmitted. The transmission data is video and / or audio stream data, and the plurality of types of segment files are a plurality of types of video and / or audio segment files having different viewpoints and / or listening points.
The client according to claim 2, wherein the data requesting unit sets the highest priority to the video and / or audio segment file related to the gazing point and / or the gazing point. For each of the plurality of types of segment files, a plurality of segment files having different data amounts are prepared.
The data requesting means determines, based on the received instruction or a predetermined item, which segment file at that stage is selected for each segment file to be acquired when receiving the next segment. The client according to any one of claims 1 to 3, wherein: For each of the plurality of types of segment files, a plurality of segment files having different data amounts are prepared.
When the data request means receives one segment, it determines the stage of the segment file to be requested next or thereafter based on the reception status of the segment file received as a result of one request. The client according to any one of claims 1 to 4, wherein: The transmission data is video and / or audio stream data, and the plurality of types of segment files are a plurality of types of video and / or audio segment files having different viewpoints and / or listening points. Segment files are video and / or audio segment files with different bit rates,
The data requesting means determines a bit rate of a segment file to be acquired next or thereafter based on a reception condition of a segment file of a certain bit rate received as a result of one request. The client according to claim 5.   The data requesting means is a bit to be selected for each of the plurality of types of video and / or audio segment files to be subsequently requested based on the throughput calculated when receiving the segment file of the certain bit rate. 7. The client according to claim 6, wherein a bit rate distribution to which a rate is assigned is determined. Reception delay determining means for determining whether reception of one segment is completed at least by the start of reception of the next segment to be received next;
The data requesting unit determines the number of segment files to be acquired in each subsequent request based on the determination result of the reception delay determining unit when receiving the one segment. The client according to any one of claims 1 to 7.   9. The data requesting unit according to claim 8, wherein when requesting acquisition of a plurality of segment files in one request to be performed thereafter, the data requesting unit includes a push transmission instruction by the server in the one request. Listed clients. A client control program that causes a computer mounted on a client to a server capable of transmitting transmission data divided into a plurality of segments and having a plurality of types of segment files prepared for each segment,
Because
In receiving one segment, a request for obtaining one or a part of the plurality of types of segment files prepared as the one segment is transmitted to the server, and then sequentially requested. Data request means for sending a request to acquire at least one of the remaining segment files to the server;
A client control program for causing a computer to function as data processing means for executing processing of a received segment file for one segment. A client control method in a computer mounted on a client for a server capable of transmitting transmission data in which a plurality of segment files are prepared for each segment, the transmission data being divided into a plurality of segments,
In receiving one segment, a request for obtaining one or a part of the plurality of types of segment files prepared as the one segment is transmitted to the server, and then sequentially requested. Sending a request to obtain at least one of the remaining segment files to the server;
And a step of executing processing of a segment file for one received segment.

Images