JP4094942B2 - Arbitrary viewpoint image transmission method, apparatus for implementing the method, processing program therefor, and recording medium - Google Patents

Description

【０１３４】
ここで、[x,y,z]^Tは３次元デカルト座標系における仮想視点位置、[x,y,z]^Tの変数x,y,zに記号“・”が付けられたものは、その速度を表している。このとき、状態量の最適な推定値と一期先予測値は、以下に述べるカルマンフィルタにより計算することができる。
【０１３５】
まず、システムモデルは数２で記述できるものとする。
【数２】

【０１３６】
ここで、ｗ_tは系に加わる６次元の外乱ベクトルで、平均が０、共分散がＱ_t、で既知であるものとする。また、Ｆは状態の推移行列を表し、次の６×６の行列であるものとする。
【０１３７】
【数３】

【０１３８】
一方、時刻tでの観測値は次の数４で表される。
【数４】

ここで、Ｈは６×６の単位行列とし、ｖ_tは６次元の推定誤差ベクトルで、平均が０、共分散がＲ_tで既知であるものとする。
【０１３９】
ｗ_t、ｖ_tは共に独立な離散化された正規白色雑音で、状態量Ｘ_tとも独立である場合、以上のモデルによるカルマンフィルタのアルゴリズムは、以下の様になる。
【０１４０】
すなわち、
［フィルタリング］
【数５】

【数６】

【数７】

【０１４１】
［一期先予測］
【数８】

【数９】

【０１４２】
［初期値］
【数１０】

【０１４３】
数５〜数７により、状態量の最適な推定値更新処理を行う。また、一期先予測値を数８及び数９により計算する。
【０１４４】
以上がカルマンフィルタによる視点移動の推定法であるが、本発明は、このアルゴリズムを用いた推定法に限定されるものではないものとする。
【０１４５】
以上説明した様に本実施形態の任意視点画像伝送装置によれば、利用者の所望する視点位置の移動範囲を予測し、その範囲内の視点位置での画像生成に必要なパケットを選択するので、伝送帯域を必要最小限に節約することが可能である。
【０１４６】
【発明の効果】
本発明によれば、複数の領域の部分画像に分割された画像データの内、利用者の所望する視点位置の画像を生成するのに必要な部分画像を選択して伝送するので、実時間コミュニケーション、ライブ配信やストリーミング放送等、処理時間の要求が厳しいネットワーク・アプリケーションで利用者の所望する視点位置からの画像である任意視点画像を提供する場合において、伝送帯域を節約しつつも、利用者からの視点移動の要求に対する応答性を犠牲にすること無く、利用者の人数が増えても安定して、任意視点画像を生成する元となる複数の視点位置から操影した画像データを伝送することができる。
【図面の簡単な説明】
【図１】実施形態１の任意視点画像伝送装置の処理概要を示す図である。
【図２】実施形態１の任意視点画像伝送装置の概略構成を示す図である。
【図３】実施形態１の画像分割及びヘッダ付加処理の処理手順を示すフローチャートである。
【図４】実施形態１のパケット選択処理の処理手順を示すフローチャートである。
【図５】実施形態１のカメラと画像分割ユニットの接続の一形態を示す図である。
【図６】実施形態１の階層的なパケット選択処理の一例を示す図である。
【図７】実施形態１のOSI参照モデルにおける複数のレイヤへの通信ヘッダの付加処理の一例を示す図である。
【図８】実施形態１の通信ヘッダの付加処理を上位レイヤと下位レイヤで分割し、領域ヘッダ付加処理の前後にパケット選択処理を配置する場合の一例を示す図である。
【図９】実施形態１のパケット選択ユニット、ネットワーク、利用者端末に至る処理の流れの一例を示す図である。
【図１０】実施形態１の利用者端末からの視点位置入力を元に必要なパケットを算出し、パケット選択ユニット７０１に要求する処理の一例を示す図である。
【図１１】実施形態２の任意視点画像伝送装置の処理概要を示す図である。
【図１２】実施形態２の伝送路における利用者端末９０５の間近で行われるパケット選択処理の一例を示す図である。
【図１３】実施形態２の任意視点画像伝送装置９００及び９１０の概略構成を示す図である。
【図１４】実施形態２のパケット選択処理の処理手順を示すフローチャートである。
【図１５】実施形態３の画像の分割・符号化処理の概要を示す図である。
【図１６】実施形態３の画像の符号化における直交変換のブロックと、パケット分割における部分画像の関係の一例を示す図である。
【図１７】実施形態４のパケットの伝送期間中に移動する視点の範囲を推定し、その範囲での画像生成に必要なパケットを算出して要求する処理の処理手順を示すフローチャートである。
【符号の説明】
１０１…カメラ、１０２…画像分割ユニット、１０３…領域ヘッダ付加ユニット、１０４…通信ヘッダ付加ユニット、１０５…パケット選択ユニット、１０６…ネットワーク、１０７…利用者端末、１０８…パケット要求、２１０…任意視点画像伝送装置、２１１…ＣＰＵ、２１２…メモリ、２１３…磁気ディスク装置、２１４…入力装置、２１５…出力装置、２１６…ＣＤ−ＲＯＭ装置、２１７…通信装置、２１８…パケット要求管理テーブル、２０１…カメラ、２０２…画像分割ユニット、４０１〜４０３…階層、５０１…領域ヘッダ付加ユニット、５０２及び５０３…通信ヘッダ付加ユニット、５０４…パケット選択ユニット、６０１…領域ヘッダ付加ユニット、６０２…下位レイヤ通信ヘッダ付加ユニット、６０３…パケット選択ユニット、６０４…上位レイヤ通信ヘッダ付加ユニット、７０１…パケット選択ユニット、７０２〜７０４…利用者端末、７０６…入力パケット、７０７及び７０８…選択パケット、７０９…破棄パケット、７１０及び７１１…ポート、７１２〜７１７…中継ノード、８０１…利用者端末、８０２…必要パケット算出ユニット、８０３…パケット選択ユニット、９００及び９１０…任意視点画像伝送装置、９０１及び９０４…パケット選択ユニット、９０２及び９０３…中継ノード、９０５…利用者端末、９０８…パケット要求、１３０１…ＣＰＵ、１３０２…メモリ、１３０３…磁気ディスク装置、１３０４…入力装置、１３０５…出力装置、１３０６…ＣＤ−ＲＯＭ装置、１３０７…通信装置、１３１１…ＣＰＵ、１３１２…メモリ、１３１３…磁気ディスク装置、１３１４…入力装置、１３１５…出力装置、１３１６…ＣＤ−ＲＯＭ装置、１３１７…通信装置、１３１８…パケット要求管理テーブル、１１０１…ブロック化処理、１１０２…直交変換処理、１１０３…量子化処理、１１０４…領域ヘッダ付加処理、１２０１…ブロック、１２０２…パケット、１２０３…パケット。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an arbitrary viewpoint image transmission apparatus that transmits an image from an arbitrary viewpoint position desired by a user, and in particular, an arbitrary viewpoint position desired by a user based on a two-dimensional image captured at a plurality of viewpoint positions. The present invention relates to a technique that is effective when applied to an arbitrary viewpoint image transmission apparatus that transmits image data in an image processing system that generates an image from the image.
[0002]
[Prior art]
Recent developments in computer graphics (CG) and virtual reality (VR) technologies have made it possible to observe realistic images of subjects not only from the viewpoint where the camera is installed, but also from the viewpoint desired by the user. I came.
[0003]
Such image generation technology is generally called Image-Based Rendering (IBR), and conventional CG technology required a precise geometric model of the subject, whereas IBR is a two-dimensional image of the subject taken from many positions. Based on the image, an image in which the subject is viewed from an arbitrary viewpoint position can be generated with little or no geometric model.
[0004]
Representative examples of IBR include light field rendering, Lumigraph, and a technique called ray space theory (see, for example, Non-Patent Documents 1 to 3).
[0005]
In these methods, a two-dimensional image taken by a camera from a large number of positions is regarded as data obtained by sampling the intensity of a bundle of rays (a group of rays) passing through the lens center of the camera. Then, a procedure is performed in which the light group data is once expressed in a multidimensional space called a light space or the like, and then a light group passing through a desired viewpoint position is resampled on the light space.
[0006]
However, since such a method requires two-dimensional image data (multi-view data) taken at a large number of viewpoint positions in order to construct a ray space, the amount of data required compared to a method using a geometric model Has enormous drawbacks. For this reason, an approach is often taken in which the amount of data is compressed by performing the encoding process using the redundancy included in the multiview data.
[0007]
For example, some reference images are set, and other images are encoded with the difference, or once a three-dimensional model is constructed and the model is viewed from the viewpoint position, There are methods such as encoding a prediction error with an image (see, for example, Patent Documents 1 and 2).
[0008]
However, although the amount of data decreases to some extent when these encoding processes are performed, it is enormous compared to a normal two-dimensional image.
[0009]
Therefore, when considering use on a network, it is realistic to download a huge amount of data to the terminal on the user side over time, and then browse the data stored on the terminal, but it is distributed continuously. In an application called streaming in which incoming data is immediately viewed on a terminal, the bandwidth of the transmission path is insufficient, which is difficult to realize.
[0010]
Even if an approach is adopted in which a higher compression rate encoding method is used to fit within the transmission band, the higher the compression rate, the higher the computer load and the processing time. For this reason, a delay occurs and it is not suitable for communication or live distribution where real-time characteristics are important.
[0011]
These problems are not limited to data transmission between remote locations, and become a problem when a large amount of multi-view data is transmitted between component devices even in a local system. Therefore, the conventional method of reducing the amount of data by encoding significantly limits the types of applications and services that can be applied.
[0012]
On the other hand, as an approach different from such a method, first, a request for the viewpoint position is transmitted from the user side, an image of the viewpoint position requested on the multi-view camera side is generated, and the generated image is transmitted to the user side. A method of transmitting is also conceivable. In this case, if the compression efficiency by encoding is not taken into account, the data amount is the same as that of an image for one camera, and therefore transmission can be performed in a band necessary for normal two-dimensional image communication.
[0013]
In this way, the above method can be said to be ideal when focusing only on the aspect of the transmission band, but there is a problem in responsiveness from when the user requests the change of the viewpoint position until the change is reflected in the presented image. That is, in order for the viewpoint movement instructed by the user to be reflected in the presented image, it takes a transmission time for the data to reciprocate between the user side and the multi-lens camera side, resulting in a delay in the response to the user. The operability of the system will be greatly deceived.
[0014]
Also, it is difficult to provide an arbitrary viewpoint image to a certain number of users or more by this method. This is because the viewpoint camera required by each user is created one by one on the multi-lens camera side, and the computer load and the amount of data transmission increase by the number of people.
[0015]
[Non-Patent Document 1]
Marc Levoy, Pat Hanrahan, “Light Field Rendering”, SIGGRAPH'96 Conference Proceedings, Association for Computing Machinery, 1996, p.31-41
[Non-Patent Document 2]
Steven J. Gortler et al., “The Lumigraph”, SIGGRAPH'96 Conference Proceedings, Association for Computing Machinery, 1996, p. 43-54
[Non-Patent Document 3]
Uchiyamagai, “Realization of a virtual environment by combining CG models of live-action data based on ray space theory”, 3D Image Conference '96, 3D Image Conference '96 Executive Committee, 1996, p. 13-18
[Patent Document 1]
Japanese Patent Laid-Open No. 10-111951 (page 5-6, FIG. 6)
[Patent Document 2]
Japanese Patent Laid-Open No. 11-18091 (page 7-10, FIG. 1)
[0016]
[Problems to be solved by the invention]
In the conventional method of the conventional IBR, since the data amount of multi-view data taken at a large number of viewpoint positions in order to construct a light space becomes enormous, when considering use in a network, in streaming applications There is a problem that the bandwidth of the transmission path is insufficient and it becomes difficult to realize.
[0017]
In addition, even if an approach is taken to fit the transmission band by taking a higher compression rate encoding method, the higher the compression rate, the higher the computer load and the processing time, resulting in delays. Therefore, there is a problem that it is not suitable for communication or live distribution where importance is placed on real time.
[0018]
On the other hand, in the method of generating the image of the viewpoint position requested from the user side on the multi-lens camera side and transmitting it to the user side, it takes a transmission time when data is reciprocated between the user side and the multi-lens camera side. Therefore, there is a problem that the operability as a system is remarkably deceived.
[0019]
Also, in this method, the viewpoint images required by each user are created one by one on the multi-lens camera side, and the computer load and the data transmission amount increase by the number of users, so a predetermined number or more There is a problem that it is difficult to provide arbitrary viewpoint images to users.
[0020]
The object of the present invention is to solve the above problems, save the transmission band for transmitting image data, and increase the number of users without sacrificing the responsiveness to the viewpoint movement request from the users. The present invention also provides a technique capable of stably transmitting image data captured from a plurality of viewpoint positions that are sources for generating an arbitrary viewpoint image. Another object of the present invention is to save image transmission bandwidth at the end close to the user and to generate image data for generating an arbitrary viewpoint image without sacrificing the responsiveness of the image generation to the user's viewpoint movement. To provide a technique capable of dividing an image without reducing the compression efficiency compared to the case where the entire image is encoded, and saving the transmission band to the minimum necessary. is there.
[0021]
[Means for Solving the Problems]
The present invention relates to an arbitrary viewpoint image transmission apparatus that transmits image data taken from a plurality of viewpoint positions from which an arbitrary viewpoint image is generated. Among image data divided into partial images of a plurality of areas, a user A partial image necessary for generating an image at a desired viewpoint position is selected and transmitted.
[0022]
In the present invention, when an image (arbitrary viewpoint image) from a viewpoint position desired by a user is provided in a network application that requires strict processing time, such as real-time communication, live distribution, and streaming broadcasting, the following is performed. The image data is transmitted.
[0023]
That is, after each image taken at different viewpoint positions is further divided into partial images of a plurality of areas, the area header information indicating which area of which viewpoint position the data of which is divided is divided. To the partial image data. Then, a header of a communication protocol is added to the data to form a packet, and after selecting a packet including a partial image necessary to generate an image at a viewpoint position desired by the user from the packet, The selected packet is routed by the communication protocol header and transmitted.
[0024]
In a general IBR image generation method, when an image at a certain viewpoint position is generated by a user terminal, original images taken at a plurality of different viewpoint positions are used, but not all of them are necessary. It becomes only a part for each original image of each viewpoint.
[0025]
Therefore, the data including the area necessary for generating an image from a desired viewpoint position at a certain point in time is obtained by further dividing the original image captured from a plurality of viewpoints into partial areas by the proposed method. The transmission band can be saved as compared with the case of generating the image after transmitting all the original images.
[0026]
In addition, since the process of adding a header indicating area information to the divided partial image data is very simple, the processing load is very small and little time overhead occurs.
[0027]
On the other hand, since the selection processing can be determined by the header portion regardless of the content of the partial image, it can be processed with high throughput by hardware generally called a switch.
[0028]
Further, by adding a communication protocol header to the image data including the area header to form a packet, even an existing network device that does not support the area header can transmit the packet.
[0029]
Even if the number of users increases and different users need the same packet, copying such packets in the process of selection processing, without burdening the side that provides the original image, A system with excellent scalability can be realized.
[0030]
In the packet selection process of the present invention, packets may be selected step by step on the transmission path between the transmission side and the reception side.
[0031]
When the viewpoint position of the user changes, the partial image necessary for image generation also changes. Therefore, if the user moves the viewpoint during the packet transmission period, the packet required at the time of transmission differs from the packet required at the time of arrival at the receiving side. For this reason, if only the necessary packets are selected and transmitted at the time of transmission, when the user moves the viewpoint, the packets necessary for image generation may not be acquired on the receiving side. May not be able to present a correct image, and the responsiveness to the user's operation may be impaired.
[0032]
On the other hand, when packets are transmitted excessively without strict selection of packets at the time of transmission, newly necessary packets can be selected at the reception stage, but a wide transmission band is required. In general, the closer the network is to the end user side, the narrower the band, so the amount of data that could be transmitted over a wide band on the route close to the transmitting side may not be transmitted to the receiving user side. .
[0033]
However, if the packet is roughly selected on the transmission side and the packet selection is made stricter sequentially as it approaches the user by relaying at a node on the route, the transmission bandwidth at the end close to the user is saved. be able to. And the closer the user is to the node, the shorter the time it takes for the packet to reach the user, so even if the range of movement of the viewpoint is narrow and the packet selection is strict, it is necessary when arriving at the user side. Since there is little risk of packet loss, the responsiveness of image generation to the user's viewpoint movement is not sacrificed.
[0034]
In the division into partial images according to the present invention, encoding including orthogonal transformation may be performed on a partial image generated using a minimum unit of encoding processing or a set thereof as a division unit.
[0035]
In the transform coding method generally used for coding a two-dimensional image, the compression efficiency is increased by actively using the two-dimensional correlation of the image. However, when the image is divided into partial images, the compression efficiency decreases for each data, and the data size as a whole may increase.
[0036]
However, in the transform coding method, orthogonal coding is performed by, for example, discrete cosine transform (DCT) for each partial image called a block, and therefore, by dividing the partial image for each block or each set, the entire image can be obtained. Compared to the case of encoding, the image can be divided without reducing the compression efficiency.
[0037]
Further, in the present invention, assuming that the viewpoint position desired by the user moves during the transmission period, the movement range is predicted, and a packet necessary for image generation at the viewpoint position within the range is selected. Also good.
[0038]
If the user moves the viewpoint during the packet transmission period, the packet required at the time of transmission differs from the packet required at the time of arrival at the receiving side. For this reason, if only the necessary packets are selected and transmitted at the time of transmission, the necessary packets may not be acquired on the receiving side when the user moves the viewpoint. If packet selection is roughly performed, the robustness with respect to viewpoint movement is improved, but the necessary transmission band is widened accordingly.
[0039]
Therefore, in the present invention, the viewpoint movement range of the user is specifically predicted, and a packet necessary for generating an image within the predicted viewpoint range is selected. By providing a general guideline, the transmission band can be saved to the minimum necessary.
[0040]
As described above, according to the arbitrary viewpoint image transmission apparatus of the present invention, the partial image necessary for generating the image at the viewpoint position desired by the user among the image data divided into the partial images of a plurality of regions is obtained. Selective transmission saves transmission bandwidth for image data transmission and is stable even when the number of users increases without sacrificing responsiveness to user requests for viewpoint movement. Thus, it is possible to transmit image data taken from a plurality of viewpoint positions from which arbitrary viewpoint images are generated.
[0041]
DETAILED DESCRIPTION OF THE INVENTION
(Embodiment 1)
Hereinafter, the arbitrary viewpoint image transmission apparatus according to the first embodiment that transmits image data captured from a plurality of viewpoint positions, which are sources for generating an arbitrary viewpoint image, will be described.
[0042]
FIG. 1 is a diagram showing an outline of processing of the arbitrary viewpoint image transmission apparatus of the present embodiment. As shown in FIG. 1, the arbitrary viewpoint image transmission apparatus according to the present embodiment further divides each image captured at a different viewpoint position by the camera 101 into partial images of a plurality of regions by the image dividing unit 102, and then The region header addition unit 103 adds header information indicating which region of which viewpoint position the data is to the divided partial image data. The communication header adding unit 104 adds a communication protocol header to the data to form a packet, receives the packet request 108 from the user terminal 107, and generates an image of the viewpoint position desired by the user. Then, the packet selection unit 105 selects a packet including a partial image necessary for transmission, and the selected packet is routed on the network 106 by the header of the communication protocol and transmitted to the user terminal 107.
[0043]
In FIG. 1, a camera 101 is a device that captures an image of a subject from a multi-viewpoint position, and records a subject that changes every moment, for example, as a two-dimensional image of 30 frames per second, and outputs the digital data. Note that the camera 101 may sequentially output captured image data in real time, or temporarily store the image data captured by the camera 101 in a storage device, and read and output the stored data later. It may be acceptable.
[0044]
The image division unit 102 is a unit responsible for image division processing, divides a two-dimensional image of a subject output from the camera 101 into a plurality of regions, and outputs the data as a plurality of partial images.
[0045]
The area header addition unit 103 is a header information (hereinafter, referred to as “the area image”) indicating which camera the partial image data was captured with, and which area of the partial image data divided by the image division unit 102. This is a unit to which an area header is added.
[0046]
The communication header adding unit 104 adds a header for communication protocol (hereinafter referred to as a communication header) to the partial image data to which the area header is added by the area header adding unit 103 to form a cluster of data called packets. , A unit to be sent to the network 106. Here, the communication protocol corresponds to all or part of the physical / data link layer, the network layer, and the transport layer in the OSI reference model, and the communication header corresponds to, for example, an Ethernet header, an IP header, and a TCP header. However, the present invention is not limited to these specific communication protocols.
[0047]
The packet selection unit 105 sends, to the network 106, a packet including a partial image required by the user terminal 107 which is a user's arbitrary viewpoint generation terminal among the many packets sent from the communication header addition unit 104. This is a unit that performs processing for discarding unnecessary packets based on the packet area header. It is assumed that a packet request 108 indicating a necessary packet is given from the user terminal 107 side and held by the packet selection unit 105. The information held in the packet selection unit 105 is updated when the viewpoint position of the image generated by the user terminal 107 moves and the required packet changes.
[0048]
The network 106 has a transmission medium or a relay node that becomes a packet transmission path, determines a path based on the packet communication header, and transmits the packet selected by the packet selection unit 105 to the user terminal 107.
[0049]
The user terminal 107 is a terminal device used by the user. The user terminal 107 calculates a partial image necessary for generating an image at a viewpoint position desired by the user, and sends a packet including the partial image to the packet selection unit 105. Based on the packet sent in response to the request, an image at the viewpoint position is generated and presented to the user.
[0050]
FIG. 2 is a diagram showing a schematic configuration of the arbitrary viewpoint image transmission apparatus of the present embodiment. As shown in FIG. 2, the arbitrary viewpoint image transmission apparatus 210 of this embodiment includes a CPU 211, a memory 212, a magnetic disk device 213, an input device 214, an output device 215, a CD-ROM device 216, and a communication device. 217 and a packet request management table 218.
[0051]
The CPU 211 is a device that controls the operation of the arbitrary viewpoint image transmission apparatus 210 as a whole. The memory 212 is a storage device that loads various processing programs and data for controlling the operation of the entire arbitrary viewpoint image transmission apparatus 210.
[0052]
The magnetic disk device 213 is a storage device that stores the various processing programs and data. The input device 214 is a device that performs various inputs for transmitting image data of an arbitrary viewpoint image. The output device 215 is a device that performs various outputs accompanying transmission of image data of an arbitrary viewpoint image.
[0053]
The CD-ROM device 216 is a device for reading the contents of the CD-ROM on which the various processing programs are recorded. The communication device 217 is a device that communicates with the user terminal 107 via a network such as the Internet or an intranet. The packet request management table 218 is a table that stores the port number that received the packet request 108 in association with the viewpoint position identification number in the packet request 108.
[0054]
The arbitrary viewpoint image transmission apparatus 210 according to the present embodiment can be realized by a computer and a program, and causes the computer to function as the image division unit 102, the area header addition unit 103, the communication header addition unit 104, and the packet selection unit 105. The program is recorded on a recording medium such as a CD-ROM and stored in a magnetic disk or the like, and then loaded into a memory and executed. The recording medium for recording the program may be a recording medium other than the CD-ROM. Further, the program may be used by installing it from a recording medium in a computer, or the program may be used by accessing the recording medium through a network.
[0055]
FIG. 3 is a flowchart showing a processing procedure of image division and header addition processing according to this embodiment. As shown in FIG. 3, the image division unit 102 of the present embodiment divides each piece of image data taken at different viewpoint positions by the camera 101 into partial images of a plurality of regions, and then the region header addition unit 103 performs the portion. The header information indicating which region of the viewpoint position the image data is added to, and the communication header addition unit 104 adds the header of the communication protocol to the partial image data and sends it to the packet selection unit 105. .
[0056]
When the arbitrary viewpoint image transmission apparatus 210 of this embodiment receives a request for transmission of an arbitrary viewpoint image from the user terminal 107, the image division unit 102 first starts processing of the image division unit 102. A camera identification number for identifying which camera was photographed and an area identification number for identifying the divided partial image data are initialized.
[0057]
In step 302, after receiving the image data of the subject output from the camera 101, in step 303, the received image data is temporarily stored in the memory 212. Here, the image data may be read from a storage device that temporarily stores the image data captured by the camera 101 and temporarily stored in the memory 212.
[0058]
In step 304, the size and reproduction time of the image data stored in the memory 212 as described above are checked, and it is determined whether or not the stored image data has reached a division unit for dividing into partial images. If the unit has been reached, the process proceeds to step 305.
[0059]
In step 305, the region header adding unit 103 adds a region header including the viewpoint identification number and the region identification number to the partial image data in the division unit, and then increments the region identification number and proceeds to step 306.
[0060]
In step 306, the communication header adding unit 104 generates a packet by adding a communication header for transmission to the user terminal 107 to the partial image data to which the region header is added. In step 307, the generated packet is transferred to the packet selection unit 105.
[0061]
In step 308, it is checked whether or not an instruction to end image division and header addition processing has been input and whether or not processing of all original images has been completed. If it still exists, the process returns to step 302 to divide the next image data.
[0062]
When image division and header addition processing are performed on the images at the respective viewpoint positions captured by the plurality of cameras 101 as described above, the images at the respective viewpoint positions are divided into a plurality of partial images, Packets of a plurality of partial images at the viewpoint position are transferred to the packet selection unit 105 in the order of division.
[0063]
FIG. 4 is a flowchart showing the processing procedure of the packet selection processing of this embodiment. As shown in FIG. 4, the packet selection unit 105 of this embodiment receives packets of a plurality of partial images at a plurality of viewpoint positions as described above, and generates an image at a viewpoint position desired by the user. A packet including a partial image necessary for transmission is selected and transmitted.
[0064]
In step 401, the packet selection unit 105 checks whether or not the packet request 108 indicating the viewpoint position desired by the user is received from the user terminal 107. If the packet request 108 is received, the process proceeds to step 402. .
[0065]
In step 402, the viewpoint position identification number is read from the packet request 108, and the port number that received the packet request 108 is associated with the read viewpoint position identification number and held in the packet request management table 218.
[0066]
In step 403, it is checked whether or not a packet is received from the communication header adding unit 104. If a packet is received, the process proceeds to step 404.
[0067]
In step 404, referring to the area header of the packet received from the communication header addition unit 104, it is determined whether the camera identification number in the area header corresponds to the viewpoint position identification number stored in the packet request management table 218. It is determined that the packet is a packet including a partial image required by the user terminal 107, and the process proceeds to step 405. Here, information indicating which viewpoint position identification number corresponds to a certain camera identification number, that is, information indicating which camera is required to generate an image at a certain viewpoint position, It is assumed that the packet selection unit 105 is set in advance.
[0068]
In step 405, a packet including a necessary partial image is transmitted to the network 106 from the port associated with the viewpoint position identification number in the packet request management table 218 and transmitted to the user terminal 107.
[0069]
The user terminal 107 receives the packet transmitted from the packet selection unit 105, refers to the area header, rearranges and synthesizes the image data in the area identification number order, generates an image at the viewpoint position, and displays it. The image is output to an output device such as a device or a speaker, and an arbitrary viewpoint image is provided to the user.
[0070]
If it is determined in step 404 that the packet does not include a partial image required by the user terminal 107, the process proceeds to step 406, where the unnecessary packet is discarded.
[0071]
As described above, in the present embodiment, an original image captured at a plurality of viewpoint positions is divided in parallel for each viewpoint position to generate partial image data, and a plurality of partial image data at the plurality of viewpoint positions is divided. Therefore, only partial image data including an area necessary for generating an image from a desired viewpoint position at a certain point in time can be transmitted, compared to the case of generating an image after transmitting all original images. Transmission bandwidth can be saved.
[0072]
For example, when five original images at the viewpoint positions 1 to 5 are transmitted and an image at the viewpoint position 3 desired by the user is generated at the user terminal 107, conventionally, the original images at the viewpoint positions 1 and 2 are If the transmission is all completed and the transmission at the viewpoint position 3 is not started, the image at the viewpoint position 3 cannot be generated by the user terminal 107. For this reason, in order to ensure the responsiveness at the user terminal 107, a wide band is required so that all images such as the viewpoint positions 1 and 2 can be transmitted within a predetermined response time.
[0073]
On the other hand, in the present embodiment, when the original images at the viewpoint positions 1 to 5 are transmitted, the partial image 1-1 at the viewpoint position 1, the partial image 2-1 at the viewpoint position 2,. −1, partial image 1-2 at viewpoint position 1, partial image 2-2 at viewpoint position 2,..., Partial image 5-2 at viewpoint position 5, and so on. Since the partial images 1-1 to 3-1 are transferred to the packet selection unit 105 after the partial images 1-1 to 3-1 are transferred, the viewpoint position is selected when the partial image 3-1 is selected and transmitted. The generation of the third image can be started at the user terminal 107. For this reason, in order to ensure the responsiveness at the user terminal 107, it is sufficient to prepare a band that can transmit the partial image such as the partial image 3-1 to the user terminal 107 within a predetermined response time. It is possible to save.
[0074]
In this embodiment, since the packet of the divided partial image is selected and transmitted, it is possible to transmit an arbitrary viewpoint image without sacrificing responsiveness to a request for moving the viewpoint from the user. .
[0075]
For example, when five original images of the viewpoint positions 1 to 5 are transmitted and the viewpoint is changed to the viewpoint position 5 while the image at the viewpoint position 3 is generated at the user terminal 107, conventionally, the viewpoint position 3 and If transmission of the original image 4 is completed and transmission of the viewpoint position 5 is not started, the viewpoint movement to the viewpoint position 5 cannot be performed at the user terminal 107.
[0076]
On the other hand, in the present embodiment, when the original images at the viewpoint positions 1 to 5 are transmitted, the partial images divided as described above are sequentially received and selected by the packet selection unit 105. After the transfer of the partial images 2 to 5-2 to the packet selection unit 105 is completed, the selected partial image is changed from the partial image 3-2 to the partial image 5-2, and the viewpoint position 3 is changed to the viewpoint position. The user can move the viewpoint to 5 at the user terminal 107. Therefore, it is possible to transmit an arbitrary viewpoint image without sacrificing responsiveness to a viewpoint movement request from a user.
[0077]
The outline of the present embodiment has been described above. Next, details of individual components will be described.
[0078]
FIG. 5 is a diagram showing one form of connection between the camera and the image division unit of the present embodiment. In FIG. 1, one camera 101 is connected to one image division unit 102 one by one. However, the present invention is not limited to such a connection form. As shown in FIG. It is also possible to take a form in which a plurality of cameras 201 are connected to the image division unit 202.
[0079]
Similarly, regarding the subsequent area header addition unit 103 and communication header addition unit 104, the units upstream and downstream of the data flow are not limited to the one-to-one connection form. .
[0080]
FIG. 6 is a diagram illustrating an example of a hierarchical packet selection process according to the present embodiment. Although only one packet selection unit 105 is shown in FIG. 1, the present invention is not limited to such a form. For example, as shown in FIG. 6, packet selection processing is divided into layers 401 to 403. In addition, it is possible to adopt a form in which stepwise selection processing is performed in each layer to reduce the load on each constituent unit.
[0081]
For example, in the packet selection process of FIG. 6, the camera identification number corresponding to the viewpoint position identification information in the packet request 108 is a necessary packet, but the camera identification number of the received packet is in the packet request 108. If the viewpoint position identification information within a certain range including the viewpoint position identification number is determined, the packet is determined to be a necessary packet, and the range is gradually narrowed as the process proceeds from the hierarchy 401 to the hierarchy 403. Process.
[0082]
FIG. 7 is a diagram showing an example of processing for adding communication headers to a plurality of layers in the OSI reference model of the present embodiment. More specifically, the communication header addition unit 104 in FIG. 1 includes an area header addition unit 501, one or more communication header addition units 502 and 503 for processing a plurality of layers in the OSI reference model, as shown in FIG. As the communication header adding units 502 and 503, which are composed of a packet selecting unit 504, for example, an Ethernet header as a lower layer and an IP header as an upper layer can be considered. However, the present invention does not depend on these specific protocols.
[0083]
FIG. 8 is a diagram illustrating an example in which the communication header addition processing according to the present embodiment is divided into an upper layer and a lower layer, and packet selection processing is arranged before and after the region header addition processing. As shown in FIG. 8, a packet selection unit 603 is arranged between the area header addition unit 601 and the lower layer communication header addition unit 602 and the upper layer communication header addition unit 604, and header addition processing in a plurality of layers is performed. Packet selection processing may be performed at an arbitrary stage.
[0084]
Next, the flow of processing from the packet selection unit 105, the network 106, and the user terminal 107 in FIG. 1 will be described in detail with reference to FIG.
[0085]
FIG. 9 is a diagram illustrating an example of a flow of processing from the packet selection unit, the network, and the user terminal according to the present embodiment. 9, a packet selection unit 701 is a unit corresponding to the packet selection unit 105 in FIG. 1, user terminals 702 to 704 are user terminals, and relay nodes 712 to 717 are apparatuses that relay packets on the network 106. The input packet 706 represents a packet input to the packet selection unit 701, the selection packets 707 and 708 represent packets selected by the packet selection unit 701, and the discard packet 709 represents a packet discarded by the packet selection unit 701.
[0086]
The packet selection unit 701 has one input port and two output ports 710 and 711, and a user terminal 702 and a user terminal 703 are connected to the port 710 of the two output ports on the extension of the network 106. A user terminal 704 is connected to the extension of the port 711.
[0087]
At this time, the packet selection unit 701 outputs, from the port 710, the selection packet 707 necessary for image generation of the user terminal 702 and the user terminal 703 among the input packets 706 arriving at the input port every moment. The packet is transmitted to each terminal via the relay nodes 712 to 715, and similarly, the selection packet 708 necessary for the image generation of the user terminal 704 is output from the port 711, thereby passing through the relay nodes 716 and 717 of the network 106. Then, the selection packet 708 is transmitted to the user terminal 704.
[0088]
At this time, the packet selection unit 701 performs a process of copying a packet that needs to be output from both the ports 710 and 711 and outputting the packet from both ports, and discarding a discard packet 709 that is not required by any terminal.
[0089]
FIG. 10 is a diagram illustrating an example of processing for calculating a required packet based on the viewpoint position input from the user terminal according to the present embodiment and requesting the packet selection unit 701. The packet selection process as described above is performed when the packet selection unit 701 receives a packet request 108 from the user terminals 702 to 704 indicating which packet is necessary. As shown in FIG. Can output a desired viewpoint position and pass the packet request 108 to the packet selection unit 803 via a necessary packet calculation unit 802 that calculates a packet necessary for image generation at the viewpoint position. . At this time, the installation position of the necessary packet calculation unit 802 may be on the user terminal 801 side, the packet selection unit 803 side, or an arbitrary network node with respect to the network 106.
[0090]
Of the selection packet 707 output from the port 710, a packet necessary for both the user terminal 702 and the user terminal 703 is copied by the relay node 713 in the route, and the packet selection unit 701-relay node 712- It is also possible to transmit while saving the bandwidth of the route of the relay node 713, copied in advance by the packet selection unit 701, output redundantly from the port 710, branched at the relay node 713, and sent to both terminals It is also possible to transmit.
[0091]
As described above, in this embodiment, even when the number of users increases and different users need the same packet, the original image can be copied by copying such a packet in the process of selection processing and relay processing. A system with excellent scalability can be realized without imposing a burden on the provider.
[0092]
The packet processing process from the packet selection unit to the user terminal has been described above using FIG. 9 as an example, but the present invention is not limited to such a connection form.
[0093]
As described above, according to the arbitrary viewpoint image transmission apparatus of the present embodiment, a part necessary for generating an image at a viewpoint position desired by a user among image data divided into partial images of a plurality of regions. Since images are selected and transmitted, even when the number of users increases without saving transmission bandwidth for transmitting image data and without sacrificing responsiveness to requests for viewpoint movement from users It is possible to stably transmit image data captured from a plurality of viewpoint positions from which an arbitrary viewpoint image is generated.
[0094]
(Embodiment 2)
The arbitrary viewpoint image transmission apparatus according to the second embodiment that performs packet selection processing stepwise on the transmission path will be described below.
[0095]
FIG. 11 is a diagram showing an outline of processing of the arbitrary viewpoint image transmission apparatus of this embodiment. In FIG. 11, when image data is transmitted from the arbitrary viewpoint image transmission apparatus 900 that performs image division processing and header addition processing to the user terminal 905 via the relay nodes 902 and 903, two packets of the arbitrary viewpoint image transmission apparatus 910 are transmitted. This shows that the selection units 901 and 904 perform packet selection processing step by step. Here, the stepwise selection process means that a rough selection process is once performed by the packet selection unit 901, and then a narrower selection process is performed by the packet selection unit 904.
[0096]
Here, as shown in FIG. 11, a one-way transmission delay from the user terminal 905 to the packet selection unit 904 is t1, and similarly, a one-way transmission delay to the packet selection unit 901 is t2 (t2> t1). If the delays are equal, the packet that reaches the user terminal 905 at time t0 is a packet selection unit at time t0-t2 based on the packet request 908 at time t0-2 × t2 from the user terminal 905. Among the packets selected at 901, the packet selection unit 904 selected at time t0-t1 based on the packet request 908 at time t0-2 × t1 from the user terminal 905.
[0097]
If the time lag from when the user terminal 905 issues a packet request 908 to the packet selection unit 904 and the packet selection unit 901 until the actual image generation processing is set to Δt1 and Δt2, respectively, Δt1 = 2 × t1, Since Δt2 = 2 × t2 and Δt1 <Δt2, the packet selection unit 904 closer to the user terminal 905 has less time lag than the packet selection unit 901 farther from the user terminal 905, and therefore, during the time lag period. Therefore, it is difficult to cause a situation in which a packet necessary for generating an image is discarded even if packet selection is made more strict.
[0098]
Therefore, even if the packet selection unit 901 and the packet selection unit 904 perform the stepwise selection process, the performance of the responsiveness to the viewpoint movement is not deteriorated as compared with the case where the selection process is performed only by the packet selection unit 901. . On the other hand, by further narrowing down by the packet selection unit 904, a necessary packet can be sufficiently transmitted even on the transmission path on the user terminal 905 side generally having a small transmission band.
[0099]
In the above example, there are two packet selection units, but it is also possible to select packets stepwise in the transmission path by two or more packet selection units.
[0100]
FIG. 12 is a diagram illustrating an example of a packet selection process performed near the user terminal 905 in the transmission path according to the present embodiment. In this embodiment, the transmission path (backbone) up to the user terminal 905 has a sufficient transmission band to transmit all packets, and the transmission path close to the user terminal 905 has a limited transmission band. If not, an embodiment in which the packet selection unit is not installed at the entrance of the transmission path as shown in FIG. 12 and the packet selection unit is installed only on the side closer to the user of the transmission path is possible. .
[0101]
FIG. 13 is a diagram illustrating a schematic configuration of arbitrary viewpoint image transmission apparatuses 900 and 910 according to the present embodiment. As shown in FIG. 13, the arbitrary viewpoint image transmission apparatus 900 of this embodiment includes a CPU 1301, a memory 1302, a magnetic disk device 1303, an input device 1304, an output device 1305, a CD-ROM device 1306, and a communication device. 1307.
[0102]
The CPU 1301 is a device that controls the operation of the arbitrary viewpoint image transmission apparatus 900 as a whole. The memory 1302 is a storage device that loads various processing programs and data for controlling the operation of the entire arbitrary viewpoint image transmission apparatus 900.
[0103]
A magnetic disk device 1303 is a storage device for storing the various processing programs and data. The input device 1304 is a device that performs various inputs for executing image division, region header addition, and communication header addition.
[0104]
The output device 1305 is a device that performs various outputs accompanying image division, region header addition, and communication header addition. A CD-ROM device 1306 is a device for reading the contents of a CD-ROM in which the various processing programs are recorded. The communication device 1307 is a device that communicates with the arbitrary viewpoint image transmission device 910 via a network such as the Internet or an intranet.
[0105]
The arbitrary viewpoint image transmission device 910 includes a CPU 1311, a memory 1312, a magnetic disk device 1313, an input device 1314, an output device 1315, a CD-ROM device 1316, a communication device 1317, and a packet request management table 1318. Have.
[0106]
The CPU 1311 is a device that controls the operation of the arbitrary viewpoint image transmission device 910 as a whole. The memory 1312 is a storage device that loads various processing programs and data for controlling the operation of the entire arbitrary viewpoint image transmission apparatus 910.
[0107]
The magnetic disk device 1313 is a storage device that stores the various processing programs and data. The input device 1314 is a device that performs various inputs for executing packet selection. The output device 1315 is a device that performs various outputs associated with packet selection.
[0108]
A CD-ROM device 1316 is a device for reading the contents of a CD-ROM in which the various processing programs are recorded. The communication device 1317 is a device that communicates with the arbitrary viewpoint image transmission device 900, the relay nodes 902 and 903, and the user terminal 905 via a network such as the Internet or an intranet. The packet request management table 1318 is a table that stores the port number that received the packet request 908 in association with the viewpoint position identification number in the packet request 908.
[0109]
The arbitrary viewpoint image transmission apparatus 900 of the present embodiment can be realized by a computer and a program. A program for causing a computer to function as the image division unit 102, the area header addition unit 103, and the communication header addition unit 104 is a CD- It is assumed that after being recorded on a recording medium such as a ROM and stored on a magnetic disk or the like, it is loaded into a memory and executed. The recording medium for recording the program may be a recording medium other than the CD-ROM. Further, the program may be used by installing it from a recording medium in a computer, or the program may be used by accessing the recording medium through a network.
[0110]
The arbitrary viewpoint image transmission apparatus 910 according to the present embodiment can also be realized by a computer and a program. A program for causing the computer to function as the packet selection unit 901 or the packet selection unit 904 is stored on a recording medium such as a CD-ROM. It is assumed that after being recorded and stored in a magnetic disk or the like, it is loaded into a memory and executed. The recording medium for recording the program may be a recording medium other than the CD-ROM. Further, the program may be used by installing it from a recording medium in a computer, or the program may be used by accessing the recording medium through a network.
[0111]
FIG. 14 is a flowchart showing the processing procedure of the packet selection processing of this embodiment. As shown in FIG. 14, the packet selection unit of the arbitrary viewpoint image transmission apparatus 910 according to the present embodiment receives packets of a plurality of partial images at a plurality of viewpoint positions from the arbitrary viewpoint image transmission apparatus 910 or the relay node 903 in the respective division orders. A packet including a partial image necessary for generating an image at a viewpoint position desired by the user is selected and transmitted.
[0112]
In step 1401, the packet selection unit of the arbitrary viewpoint image transmission apparatus 910 checks whether or not the packet request 908 indicating the viewpoint position desired by the user has been received from the user terminal 905, and has received the packet request 908. Then, go to Step 1402.
[0113]
In step 1402, the viewpoint position identification number is read from the packet request 908, and the port number that received the packet request 908 is associated with the read viewpoint position identification number and stored in the packet request management table 1318.
[0114]
In step 1403, it is checked whether a packet is received from the communication header addition unit 104 of the arbitrary viewpoint image transmission apparatus 910 or the relay node 903. If a packet is received, the process proceeds to step 1404.
[0115]
In step 1404, referring to the area header of the packet received from the communication header adding unit 104, the camera identification number in the area header is within a predetermined range including the viewpoint position identification number stored in the packet request management table 1318. If it is, it is determined that the packet is a packet including a partial image required by the user terminal 905, and the process proceeds to step 1405. Here, the predetermined range is set to a rough range in the packet selection unit 901, and a narrowed range is set in the packet selection unit 904. For example, the viewpoint in the packet request management table 1318 is set as a rough range. It is assumed that a range of several tens of percent before and after the position identification number and a range of several percent before and after the viewpoint position identification number in the packet request management table 1318 are set as a narrowed range.
[0116]
In step 1405, the packet determined to include the necessary partial image is transmitted from the port associated with the viewpoint position identification number in the packet request management table 1318, and the packet is transmitted to the relay node 902 or the user terminal. To 905.
[0117]
When a packet is transmitted from the packet selection unit 901 to the relay node 902, the relay node 902 receives the packet from the packet selection unit 901, determines the transmission path with reference to the communication header, and then transmits the packet to the relay node. The packet is transmitted to the packet selection unit 904 via 903.
[0118]
When a packet is transmitted from the packet selection unit 904 to the user terminal 905, the user terminal 905 receives the packet from the packet selection unit 904, sorts the image data in order of region identification numbers with reference to the region header. The image is synthesized at the viewpoint position and output to an output device such as a display device or a speaker to provide an arbitrary viewpoint image to the user.
[0119]
If it is determined in step 1404 that the packet is not a packet including a partial image required by the user terminal 905, the process proceeds to step 1406 to perform processing for discarding the unnecessary packet.
[0120]
As described above, according to the arbitrary viewpoint image transmission apparatus of the present embodiment, packets are selected step by step on the transmission path, so that the transmission band at the end close to the user is saved and the viewpoint of the user is moved. It is possible to transmit image data for generating an arbitrary viewpoint image without sacrificing the responsiveness of image generation with respect to.
[0121]
(Embodiment 3)
The arbitrary viewpoint image transmission apparatus according to the third embodiment will be described below in which the minimum unit of encoding processing including orthogonal transformation or a set thereof is a division unit of partial images.
[0122]
In the present embodiment, in the image division processing in the image division unit 102 in FIG. 1, not only the image is divided but also the encoding processing is performed, and the data size of the packet is compressed, thereby reducing the bandwidth required for transmission. .
[0123]
FIG. 15 is a diagram showing an outline of image division / encoding processing according to the present embodiment. In the figure, block processing 1101 divides the entire digital image into partial images called blocks, and orthogonal transform processing 1102 performs orthogonal transform by, for example, discrete Fourier transform (DFT) or discrete cosine transform (DCT) to quantize the image. In 1103, the number of encoded bits is adaptively distributed for each frequency component to the transform coefficient after the orthogonal transform, the region header is added in the region header addition processing 1104, and the result is output to the subsequent processing.
[0124]
FIG. 16 is a diagram showing an example of the relationship between the orthogonal transform block in the image coding of this embodiment and the partial image in the packet division. In FIG. 16, reference numeral 1201 denotes one block of orthogonal transform, and an encoding process is performed for each block, and a packet like 1202 is generated by adding a region header.
[0125]
Thus, by matching the unit of image segmentation for packetization with the block in orthogonal transform, the compression efficiency of encoding does not decrease even when compared with the case where the entire image is encoded by orthogonal transform. Can be.
[0126]
Note that the orthogonal transform in the present invention is not limited to the DFT and DCT shown in the above example, and includes an orthogonal transform using other bases. In addition, the present invention includes a case where one packet is generated for a plurality of blocks of orthogonal transform, and can take a form in which packets are collected, for example, 1203 in FIG. Furthermore, the encoding process in the present invention is not limited to orthogonal transformation, and is generally applied to a method of encoding an image by dividing it into partial areas.
[0127]
As described above, according to the arbitrary viewpoint image transmission apparatus of the present embodiment, the minimum unit of encoding processing including orthogonal transformation or a set thereof is used as a division unit of partial images, so that it is compared with the case where the entire image is encoded. Thus, it is possible to divide the image without reducing the compression efficiency.
[0128]
(Embodiment 4)
An arbitrary viewpoint image transmission apparatus according to the fourth embodiment that predicts a moving range of a viewpoint position desired by a user and selects a packet necessary for image generation at a viewpoint position within the range will be described below.
[0129]
As described above, there is a time lag between the user terminal 107 requesting the packet selection unit 105 a packet necessary for image generation and the actual reception of the packet. The position can move.
[0130]
In the present embodiment, assuming a packet transmission delay and viewpoint movement between them, a range in which the viewpoint moves is predicted in advance, and packets necessary for generating an image at a viewpoint position within the range are transmitted to the user terminal. Request from 107. The packet selection unit 105 receives the packet request 108 from the user terminal 107, selects and transmits a packet necessary for image generation at the viewpoint position within the predicted range. Here, the prediction process may be performed by the packet selection unit 105.
[0131]
FIG. 17 is a flowchart illustrating a processing procedure of processing for estimating and requesting a range of viewpoints that move during a packet transmission period, and calculating and requesting a packet necessary for image generation within the range. In the figure, in step 1301, the current viewpoint position is acquired, in step 1302, the packet transmission delay Δt is obtained by actual measurement or estimation, in step 1303, the range of viewpoint movement between Δt is estimated, In 1304, a packet including a partial image of the camera necessary for generating an image in the range of the viewpoint is calculated, and the packet calculated in step 1305 is requested to the packet selection unit.
[0132]
Here, the estimation of the viewpoint movement range in step 1303 can be obtained by estimating the viewpoint position after Δt using, for example, the following Kalman filter algorithm.
[0133]
First, the state of the virtual viewpoint at time t is represented by a vector composed of six parameters shown in Equation 1.
[Expression 1]

[0134]
Where [x, y, z] ^T Is the virtual viewpoint position in the 3D Cartesian coordinate system, [x, y, z] ^T The variable “x”, “y”, “z” with the symbol “·” represents the speed. At this time, the optimum estimated value of the state quantity and the first predicted value can be calculated by the Kalman filter described below.
[0135]
First, the system model can be described by Equation 2.
[Expression 2]

[0136]
Where w _t Is a 6-dimensional disturbance vector applied to the system, the mean is 0, and the covariance is Q _t , And are known. Further, F represents a state transition matrix, and is the following 6 × 6 matrix.
[0137]
[Equation 3]

[0138]
On the other hand, the observed value at time t is expressed by the following equation (4).
[Expression 4]

Here, H is a 6 × 6 unit matrix, and v _t Is a 6-dimensional estimated error vector, with mean 0 and covariance R _t Is known.
[0139]
w _t , V _t Are both discretized normal white noises with state quantity X _t If both are independent, the algorithm of the Kalman filter based on the above model is as follows.
[0140]
That is,
[filtering]
[Equation 5]

[Formula 6]

[Expression 7]

[0141]
[Forecast]
[Equation 8]

[Equation 9]

[0142]
[default value]
[Expression 10]

[0143]
The optimal estimated value update processing of the state quantity is performed according to Equations 5 to 7. In addition, the predicted value for the first term is calculated by Equation 8 and Equation 9.
[0144]
The above is the viewpoint movement estimation method using the Kalman filter, but the present invention is not limited to the estimation method using this algorithm.
[0145]
As described above, according to the arbitrary viewpoint image transmission apparatus of the present embodiment, the movement range of the viewpoint position desired by the user is predicted, and packets necessary for image generation at the viewpoint position within the range are selected. It is possible to save the transmission band to the minimum necessary.
[0146]
【The invention's effect】
According to the present invention, since the partial image necessary for generating the image at the viewpoint position desired by the user is selected and transmitted from the image data divided into the partial images of a plurality of regions, real-time communication is performed. When providing an arbitrary viewpoint image, which is an image from a viewpoint position desired by the user, in a network application that requires strict processing time, such as live distribution or streaming broadcasting, while saving transmission bandwidth, Without sacrificing the responsiveness to the request to move the viewpoint, even if the number of users increases, image data manipulated from multiple viewpoint positions that are the basis for generating an arbitrary viewpoint image can be transmitted. Can do.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an outline of processing of an arbitrary viewpoint image transmission apparatus according to a first embodiment.
FIG. 2 is a diagram illustrating a schematic configuration of an arbitrary viewpoint image transmission apparatus according to the first embodiment.
FIG. 3 is a flowchart illustrating a processing procedure of image division and header addition processing according to the first exemplary embodiment.
FIG. 4 is a flowchart illustrating a processing procedure of packet selection processing according to the first embodiment.
FIG. 5 is a diagram illustrating one form of connection between the camera and the image division unit of the first embodiment.
FIG. 6 is a diagram illustrating an example of a hierarchical packet selection process according to the first embodiment.
7 is a diagram illustrating an example of processing for adding communication headers to a plurality of layers in the OSI reference model of Embodiment 1. FIG.
FIG. 8 is a diagram illustrating an example when the communication header addition processing according to the first embodiment is divided into an upper layer and a lower layer, and packet selection processing is arranged before and after the region header addition processing.
FIG. 9 is a diagram illustrating an example of a flow of processing from a packet selection unit, a network, and a user terminal according to the first embodiment.
FIG. 10 is a diagram illustrating an example of processing for calculating a required packet based on viewpoint position input from a user terminal according to the first embodiment and requesting the packet selection unit 701;
FIG. 11 is a diagram illustrating a processing outline of an arbitrary viewpoint image transmission apparatus according to a second embodiment.
FIG. 12 is a diagram illustrating an example of a packet selection process performed near the user terminal 905 in the transmission path according to the second embodiment.
FIG. 13 is a diagram illustrating a schematic configuration of arbitrary viewpoint image transmission apparatuses 900 and 910 according to the second embodiment.
FIG. 14 is a flowchart illustrating a processing procedure of packet selection processing according to the second embodiment.
FIG. 15 is a diagram illustrating an outline of image division / encoding processing according to the third embodiment;
FIG. 16 is a diagram illustrating an example of a relationship between an orthogonal transform block in image coding according to the third embodiment and a partial image in packet division;
FIG. 17 is a flowchart illustrating a processing procedure of processing for estimating a range of a viewpoint that moves during a packet transmission period and calculating and requesting a packet necessary for image generation in the range according to the fourth embodiment.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 101 ... Camera, 102 ... Image division unit, 103 ... Area header addition unit, 104 ... Communication header addition unit, 105 ... Packet selection unit, 106 ... Network, 107 ... User terminal, 108 ... Packet request, 210 ... Arbitrary viewpoint image Transmission device 211 ... CPU 212 ... Memory 213 ... Magnetic disk device 214 ... Input device 215 ... Output device 216 ... CD-ROM device 217 ... Communication device 218 ... Packet request management table 201 ... Camera 202: Image division unit, 401 to 403: Hierarchy, 501: Area header addition unit, 502 and 503 ... Communication header addition unit, 504 ... Packet selection unit, 601 ... Area header addition unit, 602 ... Lower layer communication header addition unit, 603 ... Packet selection unit 604 ... upper layer communication header addition unit, 701 ... packet selection unit, 702 to 704 ... user terminal, 706 ... input packet, 707 and 708 ... selection packet, 709 ... discard packet, 710 and 711 ... port, 712 717 ... Relay node, 801 ... User terminal, 802 ... Necessary packet calculation unit, 803 ... Packet selection unit, 900 and 910 ... Arbitrary viewpoint image transmission apparatus, 901 and 904 ... Packet selection unit, 902 and 903 ... Relay node, 905 ... User terminal, 908 ... Packet request, 1301 ... CPU, 1302 ... Memory, 1303 ... Magnetic disk device, 1304 ... Input device, 1305 ... Output device, 1306 ... CD-ROM device, 1307 ... Communication device, 1311 ... CPU , 1312... Memory, 13 3 ... Magnetic disk device, 1314 ... Input device, 1315 ... Output device, 1316 ... CD-ROM device, 1317 ... Communication device, 1318 ... Packet request management table, 1101 ... Blocking processing, 1102 ... Orthogonal transformation processing, 1103 ... Quantum 1104 ... area header addition processing, 1201 ... block, 1202 ... packet, 1203 ... packet.

Claims (15)

In an arbitrary viewpoint image transmission method for transmitting image data taken from a plurality of viewpoint positions, which is a source for generating an arbitrary viewpoint image,
Information indicating which viewpoint position identification number corresponds to the camera identification number, that is, information indicating which camera is necessary to generate an image at the viewpoint position is set in advance. And the process
A B-step of holding the viewpoint position identification number included in the packet request information from the user terminal in the packet request management table;
A first process of dividing each image data photographed at different viewpoint positions into partial images of a plurality of regions;
A second step of adding region header information including the camera identification number of the photographed camera and the region identification number of the partial image to the data of the divided partial images;
A third step of adding a communication protocol header to the data to which the region header information is added to form a packet;
Based on the realm header information in the camera identification number, the packet request management table held viewpoint location numbers, and the preset information of the packet, camera identification number corresponding to the viewpoint location numbers or examine whether, if applicable transmits to the user terminal in the order obtained by dividing the packet to the partial image, if not applicable any, characterized in that it comprises a fourth step of discarding the packet Viewpoint image transmission method.   The method for transmitting an arbitrary viewpoint image according to claim 1, wherein the packet is selected stepwise on a transmission path between a transmission side and a reception side.   3. The arbitrary viewpoint image according to claim 1, wherein encoding including orthogonal transformation is performed on a partial image generated by using a minimum unit of encoding processing or a set thereof as a division unit. Transmission method.   4. The moving range of a viewpoint position desired by a user is predicted, and a packet necessary for generating an image at a viewpoint position within the range is selected. Arbitrary viewpoint image transmission method. In an arbitrary viewpoint image transmission device that transmits image data taken from a plurality of viewpoint positions that are sources of arbitrary viewpoint images,
Information indicating which viewpoint position identification number corresponds to the camera identification number, that is, means for holding information indicating which camera image is necessary when generating an image of the viewpoint position;
A packet request management table that holds the viewpoint position identification number included in the packet request information from the user terminal;
An image dividing unit that divides each image data captured at different viewpoint positions into partial images of a plurality of regions;
An area header adding unit for adding area header information including the camera identification number of the photographed camera and the area identification number of the partial image to the data of the divided partial images;
A communication header addition unit that adds a communication protocol header to the data to which the region header information is added to form a packet;
The packets realm header information in the camera identification number, based on the packet request management table held viewpoint location numbers, and the preset information, the corresponding camera identification number to the viewpoint location numbers any or examined whether, if applicable transmits to the user terminal in the order obtained by dividing the packet to the partial image, if not applicable, characterized in that it comprises a packet selection unit to discard the packet to Viewpoint image transmission device. In an arbitrary viewpoint image transmission device that transmits image data taken from a plurality of viewpoint positions that are sources of arbitrary viewpoint images,
Information indicating which viewpoint position identification number corresponds to the camera identification number, that is, means for holding information indicating which camera image is necessary when generating an image of the viewpoint position;
A packet request management table that holds the viewpoint position identification number included in the packet request information from the user terminal;
Means for receiving a packet consisting of a partial image obtained by dividing the image, and a region header information including a camera identification number of the photographed camera and a region identification number of the partial image;
The packets realm header information in the camera identification number, based on the packet request management table held viewpoint location numbers, and the preset information, the corresponding camera identification number to the viewpoint location numbers or examine whether to, if applicable is transmitted to the user terminal in the order obtained by dividing the packet to the partial image, if not applicable arbitrary viewpoint, characterized in that it comprises a packet selection unit to discard the packet Image transmission device.   7. The arbitrary viewpoint image transmission apparatus according to claim 5, wherein the packet is selected stepwise on a transmission path between a transmission side and a reception side.   The encoding according to any one of claims 5 to 7, wherein encoding including orthogonal transformation is performed on a partial image generated by using a minimum unit of encoding processing or a set thereof as a division unit. Viewpoint image transmission device.   9. The moving range of a viewpoint position desired by a user is predicted, and a packet necessary for generating an image at a viewpoint position within the range is selected. Arbitrary viewpoint image transmission device. In a program for causing a computer to function as an arbitrary viewpoint image transmission device that transmits image data taken from a plurality of viewpoint positions that are sources for generating arbitrary viewpoint images,
Information indicating which viewpoint position identification number corresponds to the camera identification number, that is, means for holding information indicating which camera image is necessary when generating an image of the viewpoint position;
A packet request management table that holds the viewpoint position identification number included in the packet request information from the user terminal;
An image dividing unit that divides each image data captured at different viewpoint positions into partial images of a plurality of regions;
An area header adding unit for adding area header information including the camera identification number of the photographed camera and the area identification number of the partial image to the data of the divided partial images;
A communication header addition unit that adds a communication protocol header to the data to which the region header information is added to form a packet;
Based on the realm header information in the camera identification number, the packet request management table held viewpoint location numbers, and the preset information of the packet, camera identification number corresponding to the viewpoint location numbers or examine whether, if applicable transmits to the user terminal in the order obtained by dividing the packet to the partial image, if not applicable, characterized in that causes a computer to function as a packet selection unit that discards the packet program. In a program for causing a computer to function as an arbitrary viewpoint image transmission device that transmits image data taken from a plurality of viewpoint positions that are sources for generating arbitrary viewpoint images,
Information indicating which viewpoint position identification number corresponds to the camera identification number, that is, means for holding information indicating which camera image is necessary when generating an image of the viewpoint position ;
A packet request management table that holds the viewpoint position identification number included in the packet request information from the user terminal;
Means for receiving a packet consisting of a partial image obtained by dividing the image, and a region header information including a camera identification number of the photographed camera and a region identification number of the partial image;
Whether the camera identification number corresponds to the viewpoint position identification number based on the camera identification number in the area header information of the packet, the viewpoint position identification number held in the packet request management table, and the preset information investigated how, if applicable transmits to the user terminal in the order obtained by dividing the packet to the partial image, if not applicable program for causing a computer to function as a packet selection unit that discards the packet .   12. The program according to claim 10, wherein the packet is selected stepwise on a transmission path between a transmission side and a reception side.   The program according to any one of claims 10 to 12, wherein encoding including orthogonal transformation is performed on a partial image generated using a minimum unit of encoding processing or a set thereof as a division unit. .   The range of movement of a viewpoint position desired by a user is predicted, and a packet necessary for generating an image at a viewpoint position within the range is selected. Program.   A computer-readable recording medium on which the program according to any one of claims 10 to 14 is recorded.

Images