JP2019197973A - Image processing device, control method thereof, program, and image processing system - Google Patents

Abstract

To provide a technology which allows an image generating device to generate a high-quality virtual viewpoint image even while each device including a plurality of imaging means for generating a virtual viewpoint image performs communication within the allowable range of the communication path thereof.SOLUTION: An image processing device for providing an image at one viewpoint among images of a plurality of viewpoints used for generating virtual viewpoint images includes: a control section performing code amount control when the total data amount of coded image data received from a first other device and coded image data obtained by lossless encoding of an image obtained by own imaging part exceeds a pre-set threshold; and a transmission section transmitting coded image data having been received and coded image data having been coded by itself to a second other device when the total data amount is equal to or less than the threshold, while transmitting coded image data obtained by code amount control performed by the control section to the second other device when the total data amount exceeds the threshold.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for transferring a captured image for generating a virtual viewpoint image.

  Recently, a technology that generates virtual viewpoint content (virtual viewpoint image) viewed from a virtual viewpoint from multiple viewpoint images obtained by installing multiple cameras at different positions and performing synchronized shooting with the multiple cameras has attracted attention. ing. According to the technology for generating virtual viewpoint content from a plurality of viewpoint images as described above, for example, in soccer or basketball, it is possible to generate highlight scenes at various viewpoint positions where an actual camera cannot enter. Compared with, it can give a high sense of presence to the user.

  On the other hand, generation and browsing of virtual viewpoint content based on a plurality of viewpoint images is performed by collecting images taken by a plurality of cameras into an image processing unit such as a server, and processing such as three-dimensional model generation and rendering by the image processing unit. This is realized by transferring to the user terminal.

  Further, in Patent Document 1, a plurality of cameras are connected to each other through an optical fiber via a pair of control units, and the image frames of each camera are accumulated in the control unit. It describes that image output expressing motion is performed.

  In a system such as Patent Document 1 that collects captured images of a plurality of cameras on a server and generates virtual viewpoint content, the network transmission load increases according to the number of cameras. Therefore, there are various methods for reducing the total amount of image information transferred to the server. For example, Patent Document 2 describes a case where the amount of image information obtained is reduced by combining a camera with a high bit depth and a camera with a low bit depth. Patent Document 3 describes a case where the amount of image information obtained is reduced by combining a high-resolution camera and a low-resolution camera.

U.S. Pat. No. 7,106,361 Japanese Patent No. 4578566 Japanese Patent No. 4578567

  However, the methods described in Patent Literature 2 and Patent Literature 3 require very complicated processing in order to generate virtual viewpoint content from images having different bit depths and resolutions. Furthermore, when a person or the like is a target of virtual viewpoint content, it is necessary to take corresponding points for the details, and when using images with different bit depths and resolutions, it is not possible to take accurate corresponding points. . For this reason, it is desirable to transfer an area such as a person who creates virtual viewpoint content without loss.

  Further, in a system such as Patent Document 1 that collects captured images of a plurality of cameras on a server and generates virtual viewpoint content, the network transmission load and the server calculation load increase according to the number of cameras. Therefore, it is also possible to extract an object for free viewpoint synthesis and transfer only the extracted image area to the server. However, although it is a part of the image, if the virtual viewpoint content is lossless encoded, the amount of transfer data increases.

  In addition, there is a demand for real-time virtual viewpoint content creation. To meet this demand, it is desirable to reduce the data transfer delay and at the same time reduce the decoding process on the server side.

  As described above, it is desirable to perform high-quality transmission of virtual viewpoint content, reduce the amount of transfer data, and reduce the decoding processing load on the server.

  The present invention has been made in view of the above problems, and each device having a plurality of imaging means for generating a virtual viewpoint image performs image generation while communicating within the allowable range of the communication path. The apparatus intends to provide a technique that makes it possible to generate a high-quality virtual viewpoint image.

In order to solve this problem, for example, an image processing apparatus of the present invention has the following configuration. That is,
An image processing apparatus for providing an image at one viewpoint position among images at a plurality of viewpoint positions used for generating a virtual viewpoint image,
Obtaining means for obtaining an image from the imaging means;
Encoding means for converting the image acquired by the acquisition means to generate lossless encoded data;
Receiving means capable of receiving one or more pieces of encoded image data from a first other device;
Control means for performing code amount control when the total data amount of the encoded image data received by the receiving means and the encoded image data encoded by the encoding means exceeds a preset threshold;
When the total data amount is equal to or less than the threshold, the encoded image data received by the receiving unit and the encoded image data obtained by encoding by the encoding unit are transmitted to the second other device, When the total data amount exceeds the threshold value, it has transmission means for transmitting the encoded image data obtained by controlling the code amount by the control means to the second other device,
The control means includes
The encoded image data received by the receiving unit and the encoded image data to be reduced in code amount among the encoded image data obtained by the encoding unit are specified, and the code amount in the specified encoded image data is determined. A specifying means for specifying a conversion coefficient to be reduced;
Reducing means for reducing the code amount of the specified transform coefficient of the encoded image data specified by the specifying means;
Until the total data amount becomes equal to or less than the threshold, the specification by the specifying unit and the reduction by the reducing unit are performed.

  According to the present invention, each device having a plurality of imaging means for generating a virtual viewpoint image communicates within the allowable range of the communication path, but the image generating device displays a high-quality virtual viewpoint image. Can be generated.

The block diagram which shows the structure of the imaging | photography system by a several camera. 6 is a flowchart illustrating transfer data creation processing according to the first embodiment. The flowchart which shows the code amount control process for transmission in 1st Embodiment. 5 is a flowchart showing code data reduction processing according to the first embodiment. The figure for demonstrating the subband produced | generated by wavelet transformation. The flowchart which shows the code data reduction process in the modification of 1st Embodiment. The figure which shows the difference in the code data deletion result in 1st Embodiment and its modification. 9 is a flowchart showing code data reduction processing according to the second embodiment. 10 is a flowchart illustrating code data reduction processing according to the third embodiment. 10 is a flowchart illustrating transfer data creation processing according to the fourth embodiment.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the present invention, and all the combinations of features described in the present embodiment are not necessarily essential to the solution means of the present invention. In addition, about the same structure, the same code | symbol is attached | subjected and demonstrated.

[First Embodiment]
FIG. 1A shows a state in which a plurality of sensor systems constituting the image processing system in the embodiment are installed so as to surround a stadium (in the illustrated case, a soccer field). FIG. 1B is a block diagram of the image processing system 100 in the embodiment. The image processing system 100 includes sensor systems 101A to 101L, an image computing server 200 that functions as a virtual viewpoint image generation device, a switching hub 150, a network 300, and an end user terminal 400.

  The end user terminal 400 is an information processing apparatus such as a personal computer, sets the position and direction of the virtual viewpoint, and transmits the information to the image computing server 200 via the network 300. Then, the image computing server 200 generates and encodes a virtual viewpoint image at the designated virtual viewpoint position and direction from the image data from the sensor systems 101A to 101L according to the information. Then, the image computing server 200 transmits the encoded image data to the end user terminal 400. The end user terminal 400 decodes the encoded image data and displays it. Communication and processing between the image computing server 200 and the end user terminal 400 are not the main point of the present invention.

  Therefore, an operation of transmitting images obtained by 12 sets of the sensor systems 101A to 101L to the image computing server 200 will be described.

  The sensor systems 101A to 101L in the image processing system 100 of the present embodiment are connected in a daisy chain (series) via a transmission line. The transmission path is not particularly limited, and is, for example, an Ethernet cable used in a network. The sensor system 101A includes a camera 111A, a camera adapter 120A that performs encoding processing of an image obtained by imaging by the camera 111A, and transfers encoded image data obtained by the encoding to the communication path 130A. The sensor systems 101B to 101L other than the sensor system 101A have the same configuration. That is, the sensor 101B located downstream of the sensor system 101A includes a camera 111B and a camera adapter 120B. The sensor system 101L located at the end also includes a camera 111L and a camera adapter 120L.

  In the present embodiment, among the 12 sets from the sensor systems 101A to 101L, if there is no particular distinction and any of them may be used, one of them is simply referred to as the sensor system 101, and the internal configuration is also described as the camera 111 and the camera adapter 120. Describe. In the embodiment, twelve sets are described as the number of sensor systems, but this is merely an example, and the number of sensor systems is not particularly limited to this. In the present embodiment, the term “image” will be described as including the concept of a moving image and a still image unless otherwise specified. That is, the image processing system 100 according to the present embodiment can process both still images and moving images. In this embodiment, an example in which the virtual viewpoint content provided by the image processing system 100 is a virtual viewpoint image will be described, but the present invention is not limited to this. Although an example in which virtual viewpoint sound is included will be mainly described, for example, a microphone may be included in the sensor system 101 and an image and sound may be included as virtual viewpoint content. However, since the voice is not the main point of the present invention, the description in the present embodiment is omitted.

  As shown in FIG. 1A, the image processing system 100 according to the embodiment includes a plurality of sensor systems 101 for photographing a subject from a plurality of directions. The sensor systems 101 are connected by a daisy chain. This connection form has the effect of reducing the number of connection cables and saving labor in wiring work in increasing the capacity of image data as the resolution and frame rate of captured images increase to 4K or 8K. It is clearly stated here.

  However, the present invention is not limited to this, and as a connection form, a star-type network configuration in which each sensor system 101A to 101L is connected to the switching hub 150 and transmits and receives data between the sensor systems 101 via the switching hub 150 may be employed. .

  1A shows a configuration in which all of the sensor systems 101A to 101L are cascade-connected so as to form a daisy chain, but this is not a limitation. For example, the plurality of sensor systems 101 may be divided into several groups, and the sensor systems 101 may be daisy chain connected in divided groups. Then, the camera adapter 120 serving as the end of the division unit may be connected to the switching hub and input an image to the image computing server 200.

  The camera adapter 120 in the sensor system 101 receives and holds various setting information received from upstream devices (the image computing server 200 for the camera adapter 120A and the sensor system 101A for the camera adapter 120B), and performs setting processing. .

  In addition, the camera adapter 120 separates a subject that is a target of the virtual viewpoint image from an image obtained by photographing with the camera 111. Then, the camera adapter 120 encodes the separated subject image and transfers the encoded image data obtained by the encoding to a downstream apparatus.

  For example, the camera adapter 120A separates a subject that is a target of the virtual viewpoint image from an image obtained by imaging with the camera 111A, and encodes the separated image. Then, the camera adapter 120A transfers the encoded image data to the downstream sensor system 101B via the transmission path 130A.

  Similar to the camera adapter 120A of the sensor system 101A, the camera adapter 120B of the sensor system 101B separates and encodes the subject from the image captured by the camera 111B. Then, the camera adapter 120B transfers the encoded image data and the encoded image data received from the sensor system 101A to the sensor system 101C located downstream via the communication path 130B. The same applies to the sensor system 101C and thereafter.

  That is, as shown in FIG. 1A, only the encoded image data A created by the sensor system 101A is transferred between the sensor systems 101A and 101B. The encoded image data A and B created by the sensor systems 101A and 101B are transferred between the sensor systems 101B and 101C. Similarly, three encoded image data A, B, and C created by the sensor systems 101A to 101C are transferred between the sensor systems 101C and 101D. Therefore, the 11 encoded image data A, B,..., K created by the sensor systems 101A to 101K reach the sensor system 101L located at the end of the daisy chain. Then, the sensor system 101L includes 12 encoded image data A, B,... Obtained by adding the encoded image data L created by itself to the 11 encoded image data A to K transferred from the sensor system 101K. , L will be transferred to the image computing server 200.

  In the present embodiment, the camera 111 and the camera adapter 120 are separated from each other, but they may be integrated in the same casing.

  Next, the configuration and operation of the image computing server 200 will be described. The image computing server 200 according to the present embodiment is configured with storage devices such as a CPU, a ROM, a RAM, a network I / F, and an HDD, and is an information processing apparatus represented by a so-called personal computer. is there. When the power is turned on, the CPU functions as an information processing apparatus by loading an OS (operating system) from the HDD into the RAM and executing it. Further, the CPU functions as the image computing server 200 by loading a server program from the HDD into the RAM and executing it under the OS.

  The image computing server 200 performs processing on the encoded image data A to L acquired from the sensor system 101L. Therefore, the image computing server 200 includes a front-end server 210, a database 220 (hereinafter also referred to as DB), a back-end server 230, and a time server 240. In the embodiment, one device functions as the front-end server 210, the database 220, the back-end server 230, and the time server 240 by executing software, but these individual functions are independent. You may implement | achieve with an apparatus.

  The time server 240 has a function of distributing the time and the synchronization signal, and distributes the time and the synchronization signal to the sensor systems 101A to 101L via the switching hub 150. The camera adapters 120 </ b> A to 120 </ b> L that have received the time and the synchronization signal synchronize the image frames to be captured by Genlocking the cameras 111 </ b> A to 111 </ b> L based on the time and the synchronization signal. That is, the time server 240 synchronizes the shooting timings of the plurality of cameras 111. Thereby, since the image processing system 100 can generate a virtual viewpoint image based on a plurality of captured images captured at the same timing, it is possible to suppress deterioration in the quality of the virtual viewpoint image due to a shift in the capturing timing.

  The front-end server 210 performs setting processing for the sensor system 101 and processing for writing an image acquired from the sensor system 111L in the database 220 according to the camera identifier, data type, and frame number. The setting process includes a process for notifying each of the sensor systems 101A to 101L of which sensor system is the most upstream sensor, a counter C setting process (to be described later), each layer for re-encoding, A process for setting a quantization parameter set for each subband is included.

  Next, the back-end server 230 reads a corresponding image from the database 220 based on the position and direction of the virtual viewpoint designated from the end user terminal 400, and performs a rendering process to generate a virtual viewpoint image. Then, the back-end server 230 encodes the generated virtual viewpoint image and transmits the encoded image data to the end user terminal 400.

  In the embodiment, the end user terminal 400 sets the position and direction of the virtual viewpoint, but the image computing server 200 may substitute the function. For this purpose, the image computing server 200 may be provided with a user interface (a display device and an input device for inputting instructions from the user), and an application program for setting the position and direction of the virtual viewpoint may be executed. Further, the configuration of the sensor system synchronization system and the image computing server 200 is not limited to this, and various forms are conceivable. However, since it is not the main point of the present case, a detailed description is omitted.

  As described above, in the image processing system 100, the virtual viewpoint image is generated by the back-end server 230 based on the image data based on the photographing by the plurality of cameras 111 for photographing the subject from a plurality of directions. Note that the image processing system 100 in the present embodiment is not limited to the logical configuration described above, and may be realized by a physically independent configuration.

  Next, image processing performed on the separated subject image by the camera adapter 120 will be described. There are various methods for separating the subject, such as a method for detecting an object learned in advance, a method for storing an image without a subject in advance as a background image, and a method for detecting a region different from the background image as moving object detection. However, since it is not the main point of this case, a detailed explanation is omitted.

  The camera adapter 120 performs lossless compression on the separated subject image and transfers it to the adjacent downstream camera adapter using the transmission path 130. At this time, the camera adapter 120 selects one of the two types of lossless compression methods prepared in advance and encodes the subject image. The first method is a lossless compression method for each pixel without performing DWT (discrete wavelet transform) on the subject image (hereinafter referred to as first lossless compression). The second is a method in which discrete wavelet transform (DWT) is performed a predetermined number of times on the entire subject image and the wavelet transform coefficient (hereinafter referred to as DWT coefficient) is lossless compressed (hereinafter referred to as second lossless compression). is there.

  In the first lossless compression, for example, Golomb encoding is performed on a pixel basis in raster scan order, for example. Since the first lossless compression does not perform DWT processing (the number of executions of DWT can be 0), the processing load related to encoding and decoding processing is very small. However, it can be said that the first lossless compression is not suitable for code amount control. Therefore, in the present embodiment, the encoded image data generated by the first lossless compression is not a code amount control target.

  On the other hand, since the second lossless compression performs DWT processing, the processing load is larger than that of the first lossless compression. However, the encoded image data obtained by the second lossless compression process can be easily changed to lossy encoded image data by adjusting the quantization parameter. Therefore, the encoded image data obtained by the second lossless compression is a code amount control target. In the first encoding process, DWT conversion is indispensable. However, when lossless encoded image data once generated is changed to lossy encoded data, it is only necessary to restore the DWT coefficient and then quantize the data. Only entropy coding (eg Golomb coding) is required. That is, it can be said that the processing involved in changing the lossless encoded data generated by the second lossless compression to the lossy encoded data has a small processing load because the DWT conversion process is eliminated.

  Next, processing relating to transfer data executed by each of the camera adapters 120A to 120L using the above-described two compression methods will be described with reference to the flowchart of FIG. In the following description, it is assumed that one of the camera adapters 120A to 120L is defined as the focused camera adapter 120 and is performed by the focused camera adapter 120. It should be understood that camera adapters other than the camera adapter 120 of interest perform the same processing.

  In step S201, the camera adapter 120 of interest acquires the initial counter T and the counter interval G from its own internal memory. These initial counter T and counter interval G are set from the image computing server 200 when the image processing system 100 is operated. The counter interval G is a value common to all the sensor systems 101, and is “3” in the embodiment. The initial counter value T is 0, 1, 2, 0, 1, 2,..., 0, 1, 2, and so on in the order of the camera adapters 120A, 120B, 120C, 120D. Is set to For example, T = 2 and G = 3 are set in the camera adapter 120C. In the embodiment, there are 12 sensor systems. Therefore, the number of camera adapters for which the initial counter T is set to 0, the number of camera adapters for which the initial counter T is set to 1, and the number of camera adapters for which the initial counter T is set to 2 are four.

  In step S202, the camera adapter 120 of interest initializes the counter C by substituting the initial counter T for the counter C as a variable secured in advance in the internal memory. As a result, the respective counters C of the camera adapters 120A, 120B,... Are set to 0, 1, 2, 0, 1, 2,.

  In step S203, the camera adapter 120 of interest waits for the timing at which shooting is started. This timing is according to the setting from the time server 240, as already described.

  When shooting is started, the camera adapter 120 of interest advances the process to step S204. In step S204, the camera adapter 120 of interest acquires an image of the subject area separated from the image obtained by imaging with the camera 111 as a frame image of interest to be compressed. That is, the target of the virtual viewpoint content is detected from the captured image, and only the detection area is acquired as the target frame image to be compressed. In this case, since the area of the subject is not limited to one area, a plurality of areas may be acquired as frame images to be compressed. However, in the present embodiment, in order to simplify the description, it is assumed that only one region is detected and used as a frame image to be compressed.

  In step S <b> 205, the camera adapter 120 of interest determines whether the counter C is “0”, so that the frame image of interest is encoded according to the first lossless compression or the second lossless compression. Determine. In the embodiment, when the counter C is “0”, the target camera adapter 120 encodes the target frame image by the first lossless compression (encoding method in which code amount control is impossible). Therefore, when the counter C is “0”, the camera adapter 120 of interest advances the process to step S206.

  When the counter C is other than “0” (when it is “1” or “2”), the camera adapter of interest performs second lossless compression of the frame image of interest (encoding method with controllable code amount). Encode with Therefore, when the counter C is other than “0”, the camera adapter 120 of interest advances the process to step S208.

  In step S206, the camera adapter 120 of interest encodes the frame image of interest according to the first lossless compression. In the embodiment, Golomb encoding is performed on the target pixel in the raster scan order. Note that since it is only necessary to be able to simply perform lossless encoding for each pixel, other methods such as obtaining a difference from the immediately preceding pixel value and applying predetermined lossless encoding (Golomb code) to the difference may be used. .

  When the process proceeds to step S208, the camera adapter 120 of interest encodes the frame image of interest according to the second lossless compression. That is, the camera adapter 120 of interest performs DWT transformation (wavelet transformation) on the subject frame image for a preset number of times to obtain a DWT coefficient. Then, the camera adapter 120 of interest performs lossless encoding of the DWT coefficient for each component. The number of DWT conversions in the embodiment is three. In addition, lossless coding of DWT coefficients uses Golomb coding, but other coding may be used.

  As is apparent from the above description, there are twelve camera adapters 120 in the image processing system of the embodiment. Therefore, four of them perform the first lossless compression process of step S206, and the remaining eight perform the second lossless compression process of step S208.

  In step S209, the camera adapter 120 of interest adds “1” to the counter C, and updates the value of the counter C with the remainder when the added value is divided by the counter interval G. That is, the value of the counter C is updated so as to circulate between 0 and G-1. Since the initial counter T of the camera adapter 120A is “0”, the counter C is updated in the order of 0, 1, 2, 0, 1, 2,... Every time one frame of encoded image data is generated. become. Further, since the initial counter T of the camera adapter 120B is “1”, the counter C is updated in the order of 1, 2, 0, 1, 2, 0... Every time one frame of encoded image data is generated. Will be.

  The above is expressed for the entire sensor systems 101A to 101L as follows.

  The counter C of each of the sensor systems 101A to 101L was 0, 1, 2, 0, 1, 2,... 2 in the first frame where the imaging was started, but in the next frame, 1, 2, 0, 1, 2, 0,... Then, at the time of the next frame shooting, 2, 0, 1, 2, 0,. Note that the value of the counter C is referred to in the subsequent code amount control as the data number of each code data.

  In step S210, the camera adapter 120 of interest performs code amount control processing based on the total amount of code data to be transferred downstream. A specific example of the code amount control process will be described later with reference to FIG.

  In step S211, the camera adapter 120 of interest outputs and transmits the data obtained by the code amount control in step S210 to the internal transmission buffer in order to transfer the data downstream. In step S212, the camera adapter 120 of interest determines whether or not an instruction indicating the end of shooting has been received from the image computing server 200. If not, the process returns to step S204. When receiving an instruction indicating the end of shooting from the image computing server 200, the camera adapter 120 ends this process.

  Next, the code amount control process in step S210 will be described with reference to the process flow of FIG.

  In order to make this processing easy to understand, numbers 0 to 11 are assigned to the camera adapters 120A to 120L in order from upstream to downstream. The encoded data generated by the camera adapter 120 indicated by the number i is represented as D (i), and the code amount is represented as S (i). The camera adapter of interest that performs the processing of FIG. 3 is assumed to be the nth (n = 0, 1,..., 11) for convenience. Then, the camera adapter 120 of interest receives the encoded image data D (0), D (1),..., D (n−1) from the upstream camera adapter. Then, the camera adapter 120 of interest generates the encoded image data D (n) of the subject image separated from the image captured by the camera 111 to which the camera adapter 120 is connected. The generated encoded data D (n) is generated in step S206 or S208 in FIG. Then, the camera adapter 120 of interest transfers the encoded image data D (0),..., D (n) downstream. Based on this point, the code amount control processing of the camera adapter 120 of interest will be described according to the flowchart of FIG.

In step S301, the camera adapter 120 of interest receives the encoded image data group Gc from the upstream through the transmission path 130, and obtains the data size Gs of the encoded image data group. Gc and Gs can be expressed as follows.
Gc = {D (0), ..., D (n-1)}
Gs = S (0) + ... + S (n-1)

  In step S302, the camera adapter 120 of interest acquires the encoded data D (n) generated by itself (code data generated in step S206 or S208) and the code amount S (n).

In step S303, the camera adapter 120 of interest calculates the total data amount TotalSize of the encoded image data to be transferred downstream. The total data amount TotalSize is obtained by the following equation.
TotalSize = Gs + S (n)

  In step S304, the camera adapter 120 of interest acquires the threshold value M indicating the upper limit size that can be transferred from the internal memory. This M is set from the image computing server 200 in this image processing system, and is a value indicating a transmission band connecting the sensor systems 101A to 101L, or a band permitted to use this system in the transmission band. is there.

  In step S305, the camera adapter 120 of interest compares the threshold value M acquired in step S304 with the total data amount TotalSize of the code data to be transmitted calculated in step S303.

  When M ≧ TotalSize, the camera adapter 120 of interest transfers the encoded data Gc received from the upstream and the lossless encoded data D (n) generated by itself without any code amount control to the downstream. It is judged that it is okay to finish this processing.

  On the other hand, if the relationship of M <TotalSize is satisfied, the camera adapter of interest advances the process to step S306 and sets the total size to M or less, so that the code amount reduction of the encoded image data D (0),. Process.

  Before describing the processing in step S306, subband data generated by DWT in the second lossless compression will be described. In general, the number of subbands obtained when DWT is performed m times is {3 × m + 1}. In the embodiment, since DWT is performed three times, ten subbands are obtained as shown in FIG.

  In the present embodiment, numbers 0, 1, 2,..., 9 are assigned in order from the low frequency subband LL. Further, the hierarchy number with the lowest resolution is set to 0, and a hierarchy number is assigned so that the hierarchy number increases by 1 as the resolution increases.

  Therefore, when the number of executions of DWT = 1, the layer number becomes two layers of 0 and 1 from the minimum resolution, subband 0 (LL) of layer number 0 and minimum resolution 0, and subband 1 (HL) of layer number 1 2 (LH) and 3 (HH) are obtained.

  When the number of executions of DWT = 3, the subband hierarchy is four. Therefore, the hierarchy number is 0, 1, 2, 3 from the minimum resolution. That is, the number of DWTs = the layer number of the maximum resolution.

  In addition, subbands required to increase the resolution by one size are collectively referred to as a hierarchy. The layer number = 0 is composed of only one subband 0 (LL), but the layer number of 1 or more is composed of high-frequency subbands {HL, LH, HH}. That is, layer number = 1 is composed of subbands 1, 2, and 3, and layer number = 2 is composed of subbands 4, 5, and 6. Therefore, the layer number = A is composed of subbands 3 × A−2, 3 × A−1, and 3 × A.

  Based on the above, the code amount control processing in step S306 will be described with reference to the flowchart of FIG. Here again, it should be noted that the camera adapter 120 of interest belongs to the nth sensor system in the sensor systems 101A to 101L where the most upstream is the 0th. It should also be noted that the camera adapter 120 of interest receives the encoded image data D (0),..., D (n-1) from the upstream and generates the encoded image data D (n) itself.

  Further, the camera adapter 120 of interest can specify the value of the counter C held by another camera adapter from the value of the counter C held by itself. For example, when the camera adapter 120 of interest is the camera adapter 120B and its counter C is 1, the counter C held by the upstream camera adapter 120A is 0, and the counter C held by the camera adapter 120C 2 and the counter C held by the camera adapter 120D can be identified as 0. In other words, the camera adapter 120 of interest receives the encoded image data D (0),..., D (n−1) received from the upstream and the encoded image data D (n) generated by the self-confidence. It is possible to determine which is encoded image data obtained in accordance with the first lossless compression and which is encoded data obtained in accordance with the second lossless compression.

  Therefore, the camera adapter of interest in the present embodiment specifies encoded image data generated according to the second lossless compression from the encoded image data D (0),... The data amount is reduced by changing the DWT coefficient of the encoded hierarchy to lossy encoding. In the embodiment, the encoded image data for reducing the data amount is selected based on the value of the counter C when the encoded data is generated. For example, among the encoded image data generated when the counter C is “1”, the encoded data of the highest hierarchy is lossy encoded to reduce the data amount. If further reduction is necessary, the encoded data of the highest layer among the encoded image data generated when the counter C is “2” is lossy-encoded to reduce the data amount. If there is still a need for further reduction, among the encoded data generated when the counter C is “1”, the encoded data one layer below the maximum layer is changed to lossy encoding. Thereafter, this process is repeated until the condition indicated by step S305 is satisfied.

FIG. 4 is a specific example of the process of step S306 of FIG. Each variable described below is secured in an internal memory (not shown) in the camera adapter 120 of interest.

In step S401, the camera adapter 120 of interest sets the maximum hierarchy number in the variable R for representing the reduction target hierarchy number. This means that the lossless code data of the layer number indicated by the variable R is converted into lossy code data. In this embodiment, since the DWT is executed three times, the variable R is initialized with “3”. In the following description, the hierarchy indicated by the variable R is simply expressed as hierarchy R.

  In step S402, the camera adapter 120 of interest acquires the quantization parameter for converting the lossless encoded data of the layer R into the lossy encoded data from the internal memory. This quantization parameter is common to the camera adapters of all sensor systems, and is stored according to an instruction from the image computing server 200.

  In step S403, the camera adapter 120 of interest initializes the variable P used for selecting the encoded image data to be reduced in data amount to “1”. In the embodiment, the encoded image data generated when the counter C = 0 is the first lossless compression and is encoded data that is not subject to code amount reduction. Therefore, if the variable P is other than “0”, it may be initialized with “2”.

  As described above, the lossless encoded data of the layer R of the encoded image data (not necessarily one) specified by the variable P among the encoded image data D (0),. The conditions for converting to data are set.

  In step S404, the camera adapter 120 of interest substitutes “0” for the variable i for specifying the encoded image data and initializes it.

  Next, in step S405, the camera adapter 120 of interest determines whether or not the value of the counter C when the encoded image data D (i) is generated matches the value indicated by the variable P. As described above, the value of the counter C when the encoded image data D (i) is generated can be determined from the value of the counter C held by the camera adapter 120 of interest. If the value of the counter C is stored in the header of the encoded image data, the determination may be made accordingly.

  When the value of the counter C when the encoded image data D (i) is generated matches the value indicated by the variable P, the encoded image data D (i) is a candidate for the code amount reduction target. Therefore, the camera adapter 120 of interest advances the process to step S406. If the value of the counter C when the encoded image data D (i) is generated does not match the value indicated by the variable P, the camera adapter 120 of interest receives the encoded image data D (i) as encoded. It is determined that it is not an amount reduction candidate, and the process proceeds to step S409.

  In step S406, the camera adapter 120 of interest checks whether the data of the layer R in the encoded image data D (i) is lossless data. Whether or not the data is lossless data is determined by referring to the quantization parameter of the layer R of the encoded image data D (i), and when it is not quantized, it is determined as lossless data. If the layer R is lossless data, the camera adapter 120 of interest determines that the code data of the layer R of the encoded image data D (i) can be reduced, and the process proceeds to step S407. If it is determined that the data in the hierarchy R is lossy data, the camera adapter 120 of interest determines that the data reduction process at this stage is not performed for the hierarchy R, and the process proceeds to step S409. Note that the situation where the data in the hierarchy R is lossy data means that the data has already been changed to lossy data in another camera adapter located upstream from the camera adapter of interest.

  In step S407, the camera adapter 120 of interest decodes the layer R of the encoded image data D (i) and restores the DWT coefficient. In step S408, the camera adapter 120 of interest quantizes the restored DWT coefficient using the quantization parameter of the layer R acquired in step S402, and recodes the quantized DWT coefficient. Turn into.

  In step S409, the camera adapter 120 of interest updates the variable i by adding “1” to the variable i. In step S410, the camera adapter 120 of interest compares the value of the variable i with the value “n” representing the order of the camera adapter of interest. If the relationship is i ≦ n, the camera adapter 120 of interest returns the process to step S405 and repeats the above-described processing until the relationship of i> n is satisfied. If the relationship i> n is satisfied, the camera adapter 120 of interest advances the process to step S411.

  As a result, the camera adapter 120 of interest re-encodes some of the encoded image data D (0),..., (N), and the code amount is reduced. Therefore, in step S411, the camera adapter 120 of interest recalculates and updates the total data amount TotalSize. In step S412, the camera adapter 120 of interest compares the updated total data amount TotalSize with the threshold value M. If the relationship of M ≧ TotalSize is satisfied, the camera adapter 120 of interest determines that the reduction process for the encoded image data to be transferred downstream has been completed, and ends this process. As a result, encoded image data D (0),..., D (n) including encoded image data with a reduced data amount is transferred downstream. On the other hand, if M <TotalSize, the camera adapter 120 of interest determines that transmission is not possible with the current amount of encoded image data, and the process proceeds to step S413.

  In step S413, the camera adapter 120 of interest adds “1” to the variable P and updates it. In step S414, it is determined whether or not the value indicated by the variable P matches the variable G indicating the counter interval. The value of the variable P takes only a value in the range of 0 to G-1. Therefore, if the variables P and G match, it means that the change of all the encoded image data obtained by the second lossless compression process to the lossy data of the layer R has been completed. Therefore, when the variables P and G match, the camera adapter 120 of interest advances the process to step S415. If the variables P and G do not match, the camera adapter 120 of interest returns the process to step S404.

  In step S415, the camera adapter 120 of interest updates the variable R by subtracting “1” from the variable R so that the hierarchy to be lossy-encoded is one level lower than before. Then, the camera adapter 120 of interest returns the process to step S402.

  As described above, according to the embodiment, by performing code amount control using hierarchical data of data obtained by encoding DWT coefficients, it is possible to control transfer time and reduce transmission delay. Furthermore, the first lossless compressed encoded image data is generated at regular intervals (3 in the embodiment) and is always transferred as lossless encoded data. Therefore, when virtual viewpoint content is created, the effect of image quality deterioration due to lossy encoding is affected. Can be reduced.

  In addition, two pieces of encoded image data whose code amount is to be reduced are selected between the logical arrangements of the sensor systems 101A to 101L. Therefore, when the sensor systems 101A to 101L are physically arranged as shown in FIG. 1A, the encoded image data of the adjacent viewpoint positions is not reduced, but the codes of the distributed viewpoint positions are reduced. Since the code amount of the converted image data is reduced, it is possible to suppress the image quality deterioration due to the code amount reduction from being concentrated on the biased viewpoint position.

  In addition, when re-encoding is performed, it is only necessary to restore the DWT coefficient (since it is not necessary to perform DWT processing), the processing load can be reduced.

  In the above-described embodiment, the number of times of performing DWT is fixed to 3 times. However, the number may be appropriately changed according to the number of camera adapters and the frame interval. For example, when the counter interval G is a small value, the number of encoded image data specified as the encoded image data group D (x) to be reduced increases, so the number of hierarchies may be small. On the other hand, when the counter interval G is a large value, the number of encoded image data specified as the encoded image data group D (x) to be reduced decreases, so the number of layers (number of executions of DWT) is set. You may increase and extend the range which can be reduced.

  Here, a comparison between the threshold M indicating the upper limit of the data transfer amount and the total data amount TotalSize in the above embodiment will be considered. It can be said that the number of encoded data received from the upstream camera adapter is smaller in the upstream camera adapter. In other words, the total amount of data tends to be less than or equal to the threshold as the camera adapter is located upstream. In other words, the probability that the total data amount exceeds the threshold increases as the camera adapter is located downstream. This means that the frequency of changing the encoded image data generated by the second lossless compression to the lossy data increases as the downstream camera adapter increases. Therefore, the threshold value M may be different for each camera adapter in order to disperse this processing load bias.

For example, in the transmission path connecting the sensor systems 101A to 101L, when the allowable band of the communication bands that can be used by this system is B and the number of sensor systems 101 is N, the image computing server 200 The front-end server 210 obtains the threshold value M for the camera adapter of the i-th (i = 0, 1,..., N−1) sensor system 101 by the following equation (1).
M = {B / N} × (i + 1) (1)
Then, the front-end server 210 causes each camera adapter to set the value M calculated for each as a threshold value. Although the threshold values M set for the camera adapters 120A to 120L are different, each camera adapter may perform processing according to the flowcharts of FIGS. As a result, it is possible to avoid the processing load due to the code amount control processing from being concentrated on the downstream side.

[Modification of First Embodiment]
In the first embodiment, the processing target in step S408 is the DWT coefficient of all subbands (three subbands HL, LH, and HH) belonging to the hierarchy R. That is, there are a relatively large number of DWT coefficients to be reduced, the amount of data reduced by the code amount reduction processing of that layer is relatively large, and the number of repetitions of the processing after S405 in FIG. However, this increases the probability that the amount of data will be reduced more than necessary, and the degree of image quality degradation will increase.

  Therefore, in the present modification, a method for reducing the data amount not in units of layers but in units of subbands will be described. The difference between this modification and the first embodiment is only the flow of the code data reduction process shown in FIG. Accordingly, the configuration of the image processing system is assumed to be the same as that of the first embodiment, and the description thereof is omitted.

  FIG. 6 is a flowchart showing a data amount reduction process in the present modification. This figure replaces FIG. 4 as described above. That is, it should be noted that the process according to FIG. 6 corresponds to the process of step S306 shown in FIG. Note that. In the process of FIG. 6, the same processes as those of the first embodiment are denoted by the same reference numerals as those of FIG.

  In step S601, the camera adapter 120 of interest substitutes and initializes the maximum subband number to the variable S for specifying the subband number targeted for data reduction. In the case of this embodiment, since the DWT is executed three times, the maximum subband number is “9” (see FIG. 5). Hereinafter, the subband indicated by the variable S is simply referred to as subband S.

  In step S602, the camera adapter 120 of interest acquires a quantization parameter for changing the lossless data of the subband S to lossy data from the internal memory.

  Steps S403 to S405 operate in the same manner as in the first embodiment.

  In step S405, the camera adapter 120 of interest determines whether or not the value of the counter C when the encoded image data D (i) is generated is the same as the value indicated by the variable P. When the value of the counter C at the time of generating the encoded image data D (i) is the same as the value indicated by the variable P, the camera adapter 120 of interest advances the process to step S603.

  In step S603, the camera adapter 120 of interest determines whether the data of the subband S of the encoded image data D (i) is lossless data. That is, if the subband S is not quantized, it is determined as lossless data. If the data of the subband S is lossless data, the camera adapter 120 of interest advances the process to step S604, and otherwise advances to step S409.

  In step S604, the camera adapter 120 of interest decodes the data of the subband S of the encoded image data D (i) and acquires the DWT coefficient of the subband S. In step S605, the camera adapter 120 of interest quantizes and re-encodes the obtained DWT coefficient using the quantization parameter acquired in the previous step S602.

  Steps S409 to S414 are the same as those in the first embodiment. In step S414, the camera adapter 120 of interest compares the value of the variable P and the counter interval G. If the two match, the process proceeds to step S606. If the two match, the process returns to S404. .

  The process proceeds to step S606 when the subband S of all the encoded image data generated when the counter C has the value indicated by the variable P is changed to lossy data. Therefore, in step S606, the camera adapter 120 of interest updates the variable S by subtracting the variable S by “1” in order to perform the code amount control of the next subband, and the process proceeds to step S602. return.

  According to the above-described modification, the code amount is reduced by converting into lossy data in the order of the subbands HH, LH, and HL of the maximum hierarchy “3”. If the reduction amount is still insufficient, the reduction process moves in the order of subbands HH, LH, and HL of the next layer “2”.

  Here, the difference in the data amount of the encoded image data according to the first embodiment and the present modification will be described with reference to the schematic diagram of FIG. In order to simplify the explanation, the figure shows a data structure of encoded image data generated by one sensor system. Reference numeral 701 in the figure indicates a state in which all subbands are lossless encoded. Reference numeral 702 indicates an example of encoded image data obtained by the code amount reduction process according to the present modification. Reference numeral 703 indicates encoded image data after the code amount reduction processing in the first embodiment described above. In the encoded image data 703 according to the first embodiment, all six subbands included in the hierarchies 1 and 2 are changed to lossy, but according to this modification, the three subbands included in the hierarchies 2 and In layer 2, only subband 3 becomes lossy. For this reason, it is obvious that transfer can be performed with better image quality than in the first embodiment.

  Also in this modification, data is reduced by the same method for all camera adapters in the sensor system, but the threshold value M set for each sensor system is set according to the equation (1) shown above. Thus, the burden on re-encoding may be distributed.

[Second Embodiment]
In the first embodiment, lossless data is converted into lossy data in units of layers. However, when taking corresponding points for generating virtual viewpoint content, it may be desirable to be able to reproduce data losslessly even at low resolution. Therefore, in the second embodiment, an example will be described in which low-resolution data is prioritized to be transferred as lossless data. The difference between the first embodiment and the second embodiment is only the code amount reduction processing in step S306 in FIG.

  Therefore, details of step S306 will be described below with reference to the flowchart of FIG. FIG. 8 is an alternative to FIG. 4 in the first embodiment. Therefore, the same processes as those in FIG. 4 are denoted by the same reference numerals, and the description thereof is omitted.

  The processing in steps S401 to S405 is the same as that in the first embodiment. In step S406, the camera adapter 120 of interest determines whether the data of the layer R of the encoded image data D (i) is lossless data. If the hierarchy R is lossless data, the camera adapter 120 of interest advances the processing to step S407 as in the first embodiment. If the data of the layer R of the encoded image data D (i) is lossy data, the camera adapter of interest advances the process to step S801.

  In step S801, the camera adapter 120 of interest deletes all the encoded data of the layer R of the encoded image data D (i). In step S <b> 802, the camera adapter 120 of interest rewrites the header information of the encoded image data D (i) to indicate that all the data of the layer R has been deleted. For example, if the code data size of each layer is described in the header, the camera adapter 120 of interest writes 0 as the code data size of the layer R. Alternatively, the image size may be rewritten to an image size that can be decoded in the hierarchy “R-1”. Finally, when the virtual viewpoint content is generated, information indicating that there is no data of the hierarchy R may be output. When the process of step S802 is completed, the camera adapter 120 of interest advances the process to step S409 and updates it by adding “1” to the variable i. Steps S410 to S414 are exactly the same operations as those in the first embodiment, and thus description thereof is omitted here.

  According to the second embodiment, compared with the first embodiment, the encoded data of the highest layer follows the process of lossless data → lossy data → deletion. As a result, the lower the resolution, the higher the probability that lossless data will be maintained.

[Third Embodiment]
In the first and second embodiments, the code data generated by all of the sensor systems 120A to 120L is obtained by uniformly reducing the image quality of the data obtained by encoding after DWT conversion, even if the image quality is low. Can reach the image computing server 200.

  However, all of the sensor systems 120A to 120L do not always operate normally. Assuming such a situation, the image computing server 200 may be equipped with a highly robust algorithm that can generate virtual viewpoint content even if code data from all sensor systems is not available. Therefore, in the third embodiment, a method of controlling the transfer data amount by thinning out data obtained by lossless encoding of DWT coefficients will be described.

  Specifically, the camera adapter 120 of interest encodes encoded image data D (i) generated by the same counter C as the value indicated by the variable P among the encoded image data D (0),..., D (n). One of these is specified. Then, the camera adapter 120 of interest deletes the encoded data in order from the highest resolution layer of the encoded image data D (i) until the total data amount TotalSize of the transmitted encoded data becomes equal to or less than the threshold value M. If even the LL subband is deleted and the encoded image data D (i) is completely deleted, but further data reduction is necessary, the camera adapter of interest will receive another data generated by the counter C indicated by the variable P. The same applies to the encoded image data. If the reduction is still insufficient, the value of the variable P is updated to specify the encoded image data to be reduced.

  The difference between the third embodiment and the first and second embodiments is the code data reduction process in step S306 in FIG. 3, and the other points are the same. FIG. 9 is a flowchart of the code data deletion process in the third embodiment. This should be understood as an alternative to FIGS.

  In step S901, the camera adapter 120 of interest substitutes “1” for the variable P for specifying the value of the counter C to be deleted and initializes it. In step S902, the camera adapter 120 of interest substitutes “0” for the variable i for specifying the encoded image data and initializes it.

  In step S903, the camera adapter 120 of interest determines whether the encoded image data D (i) has been deleted. If it has been deleted, the camera adapter 120 of interest advances the process to step S912. On the other hand, when the encoded image data D (i) exists, the camera adapter 120 of interest advances the process to step S904. In step S904, the camera adapter 120 of interest determines whether or not the value of the counter C when the encoded image data D (i) is generated matches the value indicated by the variable P. If they do not match, the camera adapter 120 of interest determines that the encoded image data D (i) is not to be deleted, and the process proceeds to step S912. If they match, the camera adapter 120 of interest determines that the encoded image data D (i) is to be deleted, and advances the process to step S905.

  In step S905, the camera adapter 120 of interest substitutes the variable R for what is the highest layer number in the encoded data included in the encoded image data D (i). In step S906, all the encoded data of the layer R of the encoded image data D (i) are deleted. In step S907, the camera adapter 120 of interest rewrites the header information of the encoded image data D (i) to indicate that all data of the layer R has been deleted. It can be said that the processes in steps S906 and S907 are the same as those in steps S801 and S802 in FIG. In step S908, the camera adapter 120 of interest calculates and updates the total data amount TotalSize again. In step S909, the camera adapter 120 of interest compares the updated total data amount TotalSize with the threshold value M. If the relationship of M ≧ TotalSize is satisfied, the camera adapter 120 of interest determines that the reduction process for the encoded image data to be transferred downstream has been completed, and ends this process. On the other hand, if the relationship of M <TotalSize is satisfied, the camera adapter 120 of interest determines that further deletion of encoded data is necessary, and advances the process to step S910.

  In step S910, the camera adapter 120 of interest subtracts “1” from the variable R and updates the variable R in order to move the hierarchy to be deleted one level down. In step S911, the camera adapter 120 of interest compares the variable R with “0”. If the value of the variable R is greater than or equal to 0, the camera adapter 120 of interest determines that there is remaining erasable encoded data in the encoded image data D (i), and returns the process to step S906. On the other hand, when the variable R is less than “0” (“−1”), the camera adapter 120 of interest has deleted the encoded data of all layers of the encoded image data D (i), that is, encoded image data. It is determined that D (i) no longer exists, and the process proceeds to step S912.

  In step S912, the camera adapter 120 of interest adds “1” to the variable i and updates the variable i. In step S913, the camera adapter 120 of interest compares the value of the updated variable i with a value “n” representing the rank of the camera adapter of interest. If i ≦ n, the camera adapter 120 of interest returns the process to step S903. If it is determined that the relationship i> n is satisfied, the camera adapter 120 of interest advances the process to step S914. In step S914, the camera adapter 120 of interest adds “1” to the variable P and updates the variable P in order to change the encoded data to be deleted. Then, the camera adapter 120 of interest returns the process to step S902.

  According to the processing described above, compared to the first and second embodiments, the number of code data to be transferred is reduced, but the lossless code data received on the server side can be increased.

  In order to generate a virtual viewpoint image, it is desirable that images of many viewpoint positions exist. On the other hand, if a certain appropriate lower limit number of viewpoint images are obtained, a virtual image of the original target image quality is obtained. A viewpoint image can be generated. In that sense, the above reduction processing may be performed up to the lower limit number.

[Fourth Embodiment]
In the first to third embodiments, out of the twelve sensor systems 101A to 101L, four sensor systems in which the counter C is 0 perform the first lossless compression, and the counter C is 1 or 2. The eight sensor system performed a second lossless compression.

  In the fourth embodiment, an example of performing second lossless compression in which all sensor systems perform DWT will be described.

  The difference between the fourth embodiment and the first embodiment is the processing related to transfer data creation in FIG. Therefore, the transfer data creation flow in the fourth embodiment will be described with reference to the flowchart of FIG.

  10 is different from FIG. 2 only in that there is no determination processing in step S205 and lossless compression of pixel values in step S206. Since others are the same as those in FIG. 2, the same reference numerals are assigned and the description thereof is omitted.

  Steps S201 to S204 are exactly the same as those in the first embodiment. The frame image acquired by the camera adapter of interest in step S204 is always subjected to the second lossless compression in step S208, and lossless encoded image data is generated. Since the processing after step S209 is exactly the same as that of the first embodiment, the description here is omitted.

  The processing in step S210 is the same as that of the first embodiment. Therefore, according to the fourth embodiment, the entire sensor system generates encoded image data according to the second lossless compression, but the counter C The encoded image data according to the second lossless compression generated when “0” is “0” is supplied to the image computing server 200 without being re-encoded.

  As described above, according to the first to fourth embodiments, the code amount of the entire system can be controlled, and at the same time, the minimum lossless data can be received on the server side. Therefore, on the server side, it is possible to create high-quality virtual viewpoint content in real time.

[Other Embodiments]
In the fourth embodiment, all the sensor systems have performed the DWT on the subject image, and described the method for reducing the data for each layer as in the first embodiment. However, data reduction in units of subbands shown in the modification of the first embodiment, data reduction in units of hierarchies shown in the second embodiment, and code data reduction shown in the third embodiment And may be combined.

  In this embodiment, the DWT coefficient is lossless compressed using Golomb code, but JPEG2000 lossless compression may be used. That is, DWT coefficients are created using 53 filters of JPEG2000, and lossless code data using DWT is created by bit-plane coding. In that case, at the time of code amount control, the code amount is reduced by deleting the data from the end of the code data of each subband. Since bit plane coding has a larger processing load than Golomb code, it is expected that the processing load of encoding on the sensor system 101 and decoding on the image computing server 200 side will be heavy. However, at the time of code amount control, it is not necessary to decode, quantize, and re-encode the DWT coefficient, and the bit plane data can be deleted without changing the code data, so the processing load at the time of code amount control is reduced.

  As described above, the characteristic processing of the embodiment is performed by the processing of the camera adapter in the sensor system. All the functions of the camera adapter may be realized by a program, or a part thereof may be realized by hardware or a circuit. When implemented as a program, the program can also be implemented by supplying the program to a system or apparatus via a network or a storage medium and reading and executing the program by one or more processors in the computer of the system or apparatus. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 ... Image processing system, 101A-101L ... Sensor system, 111A-111L ... Camera, 120A-120L ... Camera adapter, 150 ... Switching hub, 200 ... Image computing server, 300 ... Network, 400 ... End user terminal

Claims (13)

An image processing apparatus for providing an image at one viewpoint position among images at a plurality of viewpoint positions used for generating a virtual viewpoint image,
Obtaining means for obtaining an image from the imaging means;
Encoding means for converting the image acquired by the acquisition means to generate lossless encoded data;
Receiving means capable of receiving one or more pieces of encoded image data from a first other device;
Control means for performing code amount control when the total data amount of the encoded image data received by the receiving means and the encoded image data encoded by the encoding means exceeds a preset threshold;
When the total data amount is equal to or less than the threshold, the encoded image data received by the receiving unit and the encoded image data obtained by encoding by the encoding unit are transmitted to the second other device, When the total data amount exceeds the threshold value, it has transmission means for transmitting the encoded image data obtained by controlling the code amount by the control means to the second other device,
The control means includes
The encoded image data received by the receiving unit and the encoded image data to be reduced in code amount among the encoded image data obtained by the encoding unit are specified, and the code amount in the specified encoded image data is determined. A specifying means for specifying a conversion coefficient to be reduced;
Reducing means for reducing the code amount of the specified transform coefficient of the encoded image data specified by the specifying means;
The image processing apparatus characterized by performing identification by the identification unit and reduction by the reduction unit until a total data amount becomes equal to or less than the threshold value.   The image processing apparatus according to claim 1, wherein the transform performed by the encoding unit is a wavelet transform.   The encoding means applies a predetermined encoding to the reversible code without quantizing the subband wavelet transform coefficients of the resolution obtained by wavelet transform on the image obtained by the acquisition means. The image processing apparatus according to claim 2, wherein the image data is generated.   The image processing apparatus according to claim 3, wherein the specifying unit sequentially specifies encoded image data corresponding to viewpoint positions at predetermined intervals with respect to an array of a plurality of viewpoint positions. The specifying means specifies, as a reduction target, reversible wavelet transform coefficients in all subbands belonging to one resolution in the order from the maximum resolution to the minimum resolution in the specified encoded image data,
The reducing means quantizes the wavelet transform coefficient specified by the specifying means according to a preset quantization parameter, and re-encodes the quantized wavelet transform coefficient. The image processing apparatus according to claim 4. The specifying means performs reversible wavelet transform of one subband in the order from the maximum resolution to the minimum resolution in the specified encoded image data and according to a preset order belonging to one resolution. Priority is specified for reduction,
The reducing means quantizes the wavelet transform coefficient specified by the specifying means according to a preset quantization parameter, and re-encodes the quantized wavelet transform coefficient. The image processing apparatus according to claim 4. The specifying unit specifies the wavelet transform coefficients in all subbands belonging to one resolution in the order from the maximum resolution to the minimum resolution in the specified encoded image data as a reduction target,
The reduction means is
If the identified wavelet transform coefficient is reversible, quantize according to a preset quantization parameter, re-encode the quantized wavelet transform coefficient,
The image processing apparatus according to claim 4, wherein if the specified wavelet transform coefficient is irreversible, the specified wavelet transform coefficient is deleted. The specifying unit specifies the wavelet transform coefficients in all subbands belonging to one resolution in the order from the maximum resolution to the minimum resolution in the specified encoded image data as a reduction target,
The image processing apparatus according to claim 4, wherein the reducing unit deletes the specified wavelet transform coefficient.   The image processing apparatus according to claim 1, further comprising the imaging unit. The image processing apparatus is daisy chained with the first other apparatus as an upstream and the second other apparatus as a downstream,
In the case where the allowable bandwidth in the communication band of the daisy chain connection is B, the number of devices connected in the daisy chain is N, and the image processing device is the i-th device where the most upstream is 0th,
The threshold value is defined by the following formula: threshold value = {B / N} × (i + 1)
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A control method for an image processing apparatus that provides an image at one viewpoint position among images at a plurality of viewpoint positions used for generating a virtual viewpoint image,
An acquisition step of acquiring an image from the imaging means;
An encoding step of converting the image acquired in the acquisition step to generate lossless encoded data;
A receiving step capable of receiving one or more pieces of encoded image data from a first other device;
A control step for performing code amount control when the total data amount of the encoded image data received in the receiving step and the encoded image data obtained by encoding in the encoding step exceeds a preset threshold;
If the total data amount is less than or equal to the threshold, the encoded image data received in the receiving step and the encoded image data obtained by encoding in the encoding step are transmitted to the second other device, When the total data amount exceeds the threshold value, the transmission step of transmitting the encoded image data obtained by the code amount control by the control step to the second other device,
The control step includes
The encoded image data received in the reception step and the encoded image data to be reduced in code amount among the encoded image data obtained in the encoding step are specified, and the code amount in the specified encoded image data is determined. A specific process for identifying a conversion coefficient to be reduced; and
A reduction step of reducing the code amount of the specified transform coefficient of the encoded image data specified in the specifying step,
The control method for an image processing apparatus, characterized in that the specification by the specifying step and the reduction by the reduction step are performed until the total data amount becomes equal to or less than the threshold value. A computer having an imaging unit and a communication unit with another apparatus reads and executes the computer, thereby causing the computer to view one viewpoint position among images at a plurality of viewpoint positions used for generating a virtual viewpoint image. A program for functioning as an image processing apparatus that provides an image of
In the computer,
An acquisition step of acquiring an image from the imaging means;
An encoding step of converting the image acquired in the acquisition step to generate lossless encoded data;
A receiving step capable of receiving one or more pieces of encoded image data from a first other device;
A control step for performing code amount control when the total data amount of the encoded image data received in the receiving step and the encoded image data obtained by encoding in the encoding step exceeds a preset threshold;
If the total data amount is less than or equal to the threshold, the encoded image data received in the receiving step and the encoded image data obtained by encoding in the encoding step are transmitted to the second other device, When the total data amount exceeds the threshold value, a program for executing a transmission step of transmitting encoded image data obtained by code amount control in the control step to the second other device. And
The control step includes
The encoded image data received in the reception step and the encoded image data to be reduced in code amount among the encoded image data obtained in the encoding step are specified, and the code amount in the specified encoded image data is determined. A specific process for identifying a conversion coefficient to be reduced; and
A reduction step of reducing the code amount of the specified transform coefficient of the encoded image data specified in the specifying step,
The program characterized by performing the specification by the specific process and the reduction by the reduction process until the total data amount becomes equal to or less than the threshold. A plurality of imaging devices for capturing images at a plurality of viewpoint positions, and encoded data indicating the images at the plurality of viewpoint positions are received from an imaging device positioned at the end of the plurality of imaging devices, and a virtual viewpoint image is received. An image processing system configured with an image generation device to generate,
The plurality of imaging devices are connected in series,
Each of the plurality of imaging devices
Obtaining means for obtaining an image from the imaging means;
Encoding means for converting the image acquired by the acquisition means to generate lossless encoded data;
Receiving means capable of receiving one or more encoded image data transferred from upstream;
Control means for performing code amount control when the total data amount of the encoded image data received by the receiving means and the encoded image data encoded by the encoding means exceeds a preset threshold;
When the total data amount is less than or equal to the threshold, the encoded image data received by the receiving unit and the encoded image data obtained by encoding by the encoding unit are transmitted downstream, and the total data amount is A transmission means for transmitting the encoded image data obtained by controlling the code amount by the control means to the downstream when the threshold is exceeded;
The control means includes
The encoded image data received by the receiving unit and the encoded image data to be reduced in code amount among the encoded image data obtained by the encoding unit are specified, and the code amount in the specified encoded image data is determined. A specifying means for specifying a conversion coefficient to be reduced;
Reducing means for reducing the code amount of the specified transform coefficient of the encoded image data specified by the specifying means;
The image processing system characterized by performing identification by the identification unit and reduction by the reduction unit until a total data amount becomes equal to or less than the threshold value.

Images