JP2009123074A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of reducing an amount of pixel value transfer in comparison with SIMD construction and avoiding a problem of an RCSA construction (an increase in the number of cycles attributed to H. 264 block division). <P>SOLUTION: The image processing apparatus includes an array in which a plurality of arithmetic elements PE are disposed in a matrix shape. The array is divided into a plurality of sub-blocks SBSA each of which includes the prescribed number of arithmetic elements PE. Each of the plurality of sub-blocks SBSA has multiplexers 10A and 11A capable of selecting whether a self sub-block and an adjacent sub-block adjacent to the self sub-block are connected. By switching setting of the multiplexers 10A and 11A in accordance with size of an image to be processed, one or more blocks including one or more sub-blocks SBSA can be set in the array. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus for executing an image motion search.

  As a background art, a SIMD (Single Instruction Multiple Data) configuration having only an evaluation value arithmetic unit between pixels, and a RCSA (Ring Connected Systolic Array) configuration that has a pixel buffer and a high pixel reusability, Will be described. The evaluation value calculation unit is configured on the assumption that SAD (Sum of Absolute Difference) is used as the evaluation value, but other evaluation values can also be used. Also, a buffer SRAM (hereinafter referred to as “SWRAM”) storing reference image data (SW: Search Window) and a register file (hereinafter referred to as “TB buffer”) storing image data to be encoded (TB: Template Block). It is assumed that it has outside. These SWRAM and TB buffer are assumed to be capable of simultaneously outputting a maximum of 64 (= 8 × 8) pixel values in one cycle.

  An example of the SIMD configuration is shown in FIG. The SIMD configuration is a configuration having only a plurality of units for obtaining evaluation values from pixel values received from the SWRAM and the TB buffer. Since there is no buffer for holding pixel values in the SIMD configuration, pixel values cannot be reused. In an example of an RRSA (Reconfigurable Ring Connected Systlic Array) configuration according to the present invention, which will be described later, since the calculation module for the evaluation value of one pixel is 512 parallel, estimation is performed assuming that the SIMD configuration is also 512 parallel. That is, X and Y in FIG. 32 are 16 and 32, respectively. Since the maximum pixel output from the SWRAM and TB buffer is 8 × 8 pixels, for example, in order to search for one 16 × 16 point, four cycles are required while waiting for RAM reading.

  An example of the RCSA configuration is shown in FIG. The RCSA configuration is disclosed in Non-Patent Document 1 below. The RCSA configuration has a pixel value buffer in addition to an evaluation value calculation unit between pixels and is connected in a ring shape. The internal configurations of the arithmetic element PE (Processor Element) and the shift register SR (Shift Register) are shown in FIGS. 34 and 35, respectively. Similar to the SIMD configuration, the RCSA configuration is estimated as 512 parallel (PE512 parallel, SR512 parallel). That is, X and Y in FIG. 33 are 16 and 32, respectively. For example, when a 16 × 16 8-point continuous point search is performed, 4 cycles of RAM reading are performed on the PE-array side, 4 cycles of RAM reading are performed on the SR-array side, and 8 cycles are performed for evaluation value calculation. As a result, 4 + 4 + 8-1 = 15 cycles are required. Note that the reason why one cycle is subtracted lastly is that the evaluation value calculation is actually performed in the “initial load + number of shifts” cycle, so that the RAM read cycle overlaps with one evaluation value calculation. Because.

J.Miyakoshi, Y.Murachi, K.Hamano, T.Matsuno, M.Miyama and MYoshimoto, "A Low-Power Systolic Array Architecture for Block-Matching Motion Estimation," IEICE Trans. Electoronics, Vol.E88-C, No .4, pp.559-569, April 2005.

  In the SIMD configuration, the huge amount of transfer of pixel values becomes the biggest problem. Since the pixel values cannot be reused in the SIMD configuration, when performing a continuous point search, it is necessary to read out all the pixel values necessary for the calculation from the SWRAM and the TB buffer each time. As a result, a huge pixel value transfer band is required. In addition, when the number of read pixels in the RAM is limited, the cycle for reading pixel values becomes enormous in addition to the cycle for actually performing calculations. This problem relating to pixel value transfer is a problem of the SIMD configuration.

  The RCSA configuration retains reusable pixels when performing a continuous point search, thereby greatly reducing the amount of pixel value transferred from the SWRAM and TB buffer compared to the SIMD configuration. However, H. The problem with the RCSA configuration is that it does not support block division unique to H.264. That is, since all the arithmetic units can only operate synchronously even though the parallelism is 512, mode1 (16 × 16), mode2 (16 × 8), mode3 (8 × 16), and When performing a search in mode 4 (8 × 8), all 512 arithmetic units are occupied by one block of 16 × 16, 16 × 8, 8 × 16, or 8 × 8 size. Therefore, as the macroblock pair (MB-pair) to be processed is subdivided and the number of blocks increases, the number of cycles required to search for one macroblock pair increases. This H.H. The increase in the number of cycles accompanying H.264 block division becomes a problem of the RCSA configuration.

  The present invention has been made in order to solve the above-described problems in the SIMD configuration and the RCSA configuration, and can reduce the pixel value transfer amount as compared with the SIMD configuration. An object is to obtain an image processing apparatus that can avoid an increase in the number of cycles due to block division).

  An image processing apparatus according to a first aspect of the present invention includes an array in which a plurality of calculation elements for calculating an evaluation value based on pixel values of an image are arranged in a matrix, and each of the arrays has a predetermined number. The sub-block is divided into a plurality of sub-blocks including the arithmetic element, and each of the plurality of sub-blocks is selectable to select whether or not to connect the sub-block and the adjacent sub-block adjacent to the sub-block. And having one or more blocks including one or more sub-blocks in the array by switching the setting of the selection unit according to the size of the image to be processed. It is characterized by.

  The image processing apparatus according to the second invention is the image processing apparatus according to the first invention, particularly when a plurality of blocks are set in the array, each of the plurality of blocks is independent of other blocks. It is possible to operate.

  An image processing apparatus according to a third aspect of the invention is the image processing apparatus according to the first or second aspect of the invention, in particular, the sub-block includes a first unit having a plurality of arithmetic elements and the first unit in the first unit. And a second unit having a plurality of registers capable of holding pixel values calculated or calculated by the calculation element, and the selecting means uses the sub-block as an input to the first unit of the sub-block. Selection means for selecting one of the second unit of the block and the first unit of the adjacent sub-block, and as an input to the second unit of the own sub-block, the first unit of the own sub-block and the second unit of the adjacent sub-block And selecting means for selecting one.

  An image processing apparatus according to a fourth invention is the image processing apparatus according to the third invention, in particular, by connecting a plurality of the sub-blocks, so that a plurality of first units and a plurality of first blocks are included in one block. When two units are included, a register for holding a pixel value calculated or calculated by a calculation element in another first unit for a part of the plurality of first units. It can be used as a feature.

  An image processing apparatus according to a fifth aspect of the invention is an image processing apparatus according to any one of the first to fourth aspects of the invention, in particular, an image of a location adjacent to the image portion loaded in the array in a predetermined direction. A storage unit capable of holding a pixel value of a portion; and when the evaluation position of the image is shifted in the predetermined direction, the pixel value held in the storage unit is input from the storage unit to the array. Features.

  The image processing apparatus according to a sixth aspect of the present invention is the image processing apparatus according to the fifth aspect of the invention, in particular, the selection means selects one of the adjacent sub-block and the storage unit as an input to the own sub-block. It is characterized by including.

  An image processing apparatus according to a seventh invention is the image processing apparatus according to any one of the first to sixth inventions, wherein the sub-block is evaluated by the plurality of arithmetic elements in the sub-block. An adder group for adding values, the adder group for adding the evaluation values of successive rows, and for adding the evaluation value of every other row; And an adder group corresponding to the field image.

  According to the image processing apparatus according to the first to ninth inventions, the array is divided into a plurality of sub-blocks. Then, by switching the setting of the selection unit according to the size of the image to be processed, one or a plurality of blocks including one or a plurality of sub blocks are set in the array. For this reason, even if the number of blocks increases as the macro block pair to be processed increases, the number of cycles required to search for one macro block pair increases because multiple blocks set in the array can be processed simultaneously. Can be avoided.

  In particular, according to the image processing apparatus of the second invention, each of the plurality of blocks can operate independently of the other blocks, so that a plurality of blocks in one macroblock pair can be processed in parallel. . As a result, an increase in the number of cycles for searching for one macroblock pair can be avoided.

  In particular, according to the image processing apparatus of the third invention, the sub-block has the second unit having a plurality of registers that can be calculated by the calculation elements in the first unit or that can store the calculated pixel values. ing. Accordingly, since the pixel value held in the second unit can be reused, the transfer amount of the pixel value from the buffer to the sub-block can be reduced when performing the continuous point search. Further, when the sub-block and the adjacent sub-block are not connected due to the setting of the selection means, a ring-shaped path can be formed in the self-sub-block. On the other hand, when connecting the own subblock and the adjacent subblock, the first units and the second units can be connected between the own subblock and the adjacent subblock.

  In particular, according to the image processing apparatus of the fourth invention, a maximum search range that does not require an intermediate load can be expanded by treating a part of the plurality of first units in the same manner as the second unit.

  In particular, according to the image processing apparatus of the fifth invention, the pixel value of the image portion adjacent to the image portion loaded in the array is held in the storage unit, so that a snake (FS) is obtained as a FS (Full Search). A search can be executed.

  In particular, according to the image processing apparatus of the sixth aspect of the present invention, when the sub-block and the adjacent sub-block are not connected by the setting of the selection unit, a path connecting the sub-block and the storage unit is formed. Can do. On the other hand, when connecting the own subblock and the adjacent subblock, the first units and the second units can be connected between the own subblock and the adjacent subblock.

  In particular, the image processing apparatus according to the seventh aspect of the invention can handle both frame images and field images, and can handle both progressive and interlace methods. Further, it is possible to simultaneously execute the evaluation value calculation for the frame image and the evaluation value calculation for the field image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the element which attached | subjected the same code | symbol in different drawing shall show the same or corresponding element.

  FIG. 1 shows the overall configuration of an image processing apparatus according to the present invention. An image processing apparatus according to the present invention is described in H.264. It is configured as an H.264 compatible IME core 1 (IME: Integer Motion Estimation). The IME core 1 includes a reference image buffer SWRAM 2 (SWRAM: Search Window RAM), a coding target image buffer TB buffer 3 (TB: Template Block), and an image rotation processing unit (cross path). 4, an RRSA-configured array 5, and a controller 6. If these module groups are mounted in the ME core, the RRSA according to the present invention operates without any problem.

  In the IME core 1, a 2-port SRAM capable of reading out a pixel block of 8 × 1 (8 × 8), 8 × 1, and 1 × 8 size in one cycle is SWRAM2. It is installed as. Further, the SWRAM 2 can output a half pixel thinned out in the vertical direction and the horizontal direction. An image block readable from the SWRAM 2 is shown in FIG. 8 × 8 pixel block without decimation, 16 × 8 pixel block with 1/2 pixel decimation in the horizontal direction, 8 × 16 block with 1/2 pixel decimation in the vertical direction, 16 × 16 with 1/2 pixel decimation in both vertical and horizontal directions All blocks can be read in one cycle.

  The TB buffer 3 is a 512 pixel register file (or SRAM) capable of storing image data of a macroblock pair to be encoded. The TB buffer 3 can output 8 × 8, 8 × 1, and 1 × 8 size pixel blocks in one cycle from the stored image data.

  The image rotation processing unit 4 is a module for rotating an image output from the SWRAM 2 and delivering it to the data path in order to realize a vertical direction DS (Directional Search) described later. The arrangement of pixels input to the array 5 when the image is rotated is shown in FIG. It can be seen that the original image has been rotated 90 ° counterclockwise.

  The controller 6 is a module for controlling each module in the IME core 1 including the array 5. Based on the control signal from the controller 6, the array 5 performs various search operations.

  Hereinafter, the array 5 having the RRSA configuration will be described in detail. The entire configuration of the array 5 is shown in FIG. The array 5 is divided into a plurality (eight in the example shown in FIG. 4) of sub-block systolic arrays (hereinafter simply referred to as “sub-blocks”) SBSA0 to SBSA7. The sub-blocks SBSA0 to SBSA7 are configured to include processing units PU0 to PU7 and shift register units SRU0 to SRU7, respectively. Hereinafter, when subblocks SBSA0 to SBSA7 are collectively referred to as "subblock SBSA", when processing units PU0 to PU7 are collectively referred to as "processing unit PU", and when shift register units SRU0 to SRU7 are collectively referred to as "shift register" These are referred to as “unit SRU”. The processing unit PU is a unit having a plurality of arithmetic elements. The shift register unit SRU is a unit having a plurality of shift register elements for holding pixel values calculated by or calculated by the calculation elements in the processing unit PU.

  The sub-block SBSA is a systolic array module for efficiently calculating an evaluation value of 8 × 8 units. The evaluation value calculation unit is configured on the assumption that SAD (Sum of Absolute Difference) is used as the evaluation value, but other evaluation values can also be used. The sub-block SBSA includes one 8 × 8 parallel processing unit PU and one 8 × 8 parallel shift register unit SRU.

  In the sub-block SBSA, (I1) input from the SWRAM2 to the processing unit PU, (I2) input to the shift register unit SRU from the SWRAM2, (I3) laterally adjacent subblock Input to processing unit PU from processing unit PU of SBSA, (I4) Input to shift register unit SRU from shift register unit SRU of laterally adjacent sub-block SBSA, (I5) From processing unit PU of lower adjacent sub-block SBSA Input to the processing unit PU, (I6) input to the shift register unit SRU from the shift register unit SRU of the lower adjacent sub-block SBSA, (I7) to the processing unit PU from the storage unit 7 Force, and (I8) of the input to the shift register unit SRU from the storage unit 7, a total of eight input is present.

  The respective input pixel sizes are 8 × 8 pixels for (I1), 8 × 8 pixels for (I2), 1 × 8 pixels for (I3), 1 × 8 pixels for (I4), (I5) Is 8 × 1 pixels, (I6) is 8 × 1 pixels, (I7) is 8 × 1 pixels, and (I8) is 8 × 1 pixels.

  Further, in the sub-block SBSA, a bidirectional input / output path of 1 × 8 pixels exists as an internal connection between the processing unit PU and the shift register unit SRU. This path is ring-shaped. In the example shown in FIG. 4, the right output of the shift register unit SRU is the left input of the processing unit PU, the left output of the shift register unit SRU is the right input of the processing unit PU, The right output of the processing unit PU is connected to the left input of the shift register unit SRU, and the left output of the processing unit PU is connected to the right input of the shift register unit SRU.

  The sub-block SBSA outputs (O1) an output from the processing unit PU to the processing unit PU of the horizontally adjacent sub-block SBSA, and (O2) a pixel output to the outside of the sub-block. Output to shift register unit SRU, (O3) Output from processing unit PU to processing unit PU of upper adjacent subblock SBSA, and (O4) Output from shift register unit SRU to shift register unit SRU of upper adjacent subblock SBSA There are a total of four outputs.

  The respective output pixel sizes are 1 × 8 pixels for (O1), 1 × 8 pixels for (O2), 8 × 1 pixels for (O3), and 8 × 1 pixels for (O4).

  The sub-block SBSA outputs the calculation result of the evaluation value (SAD in this example) to the outside of the array 5. As shown in FIG. 6 to be described later, the SAD output from one sub-block SBSA has four Frame_4 × 4_SAD corresponding to frame images and four Field_4 × 4_SAD corresponding to field images. Depending on how these operation results are added from the eight sub-blocks SBSA0 to SBSA7, for example, mode1 to mode4 (16 × 16, 16 × 8) at the same search point during a search with a 16 × 32 macroblock pair. , 8 × 16, 8 × 8) SAD search method can be realized.

  The external input from the laterally adjacent sub-block SBSA and the internal connection between the processing unit PU and the shift register unit SRU are selectively switched depending on the operation state. Similarly, the external input from the lower adjacent sub-block SBSA and the external input from the storage unit 7 are selectively switched depending on the operation state.

  FIG. 5 shows an internal configuration of the sub-block SBSA. FIG. 6 shows an internal configuration of the evaluation value calculation part (the calculation element PE and the adder part) in the processing unit PU. FIG. 7 shows the internal configuration of the arithmetic element PE alone.

<Processing unit PU>
Referring to FIGS. 5 to 7, the processing unit PU includes an 8 × 8 pixel size data buffer, 64 (= 8 × 8) computing elements PE that perform evaluation value calculation for one pixel, and evaluation values Are added to each other according to the search state.

  Referring to FIG. 5, the processing unit PU includes a multiplexer 10A for each row of the PE matrix. The multiplexer 10A is selection means for selecting whether or not to connect the own subblock SBSA and the adjacent subblock SBSA adjacent to the own subblock SBSA. More specifically, the multiplexer 10A selects one of the shift register unit SRU of its own subblock SBSA and the processing unit PU of its adjacent subblock SBSA as an input to the processing unit PU of its own subblock SBSA. Means.

  Similarly, referring to FIG. 5, shift register unit SRU includes multiplexer 10B for each row of the SRE matrix. The multiplexer 10B is selection means for selecting whether or not to connect the own subblock SBSA and the adjacent subblock SBSA adjacent to the own subblock SBSA. More specifically, the multiplexer 10B selects one of the processing unit PU of the own subblock SBSA and the shift register unit SRU of the adjacent subblock SBSA as an input to the shift register unit SRU of the own subblock SBSA. It is a selection means.

  Depending on the size of the image to be processed, that is, depending on the macroblock pair (16 × 32), mode1 (16 × 16), mode2 (16 × 8), mode3 (8 × 16), and mode4 (8 × 8) Thus, the settings of the multiplexers 10A and 10B are switched. Further, the settings of the multiplexers 10A and 10B are switched according to the search mode to be executed, that is, depending on the FS mode, the DS mode, and the RBM mode. Details will be described later.

  As shown in FIG. 6, the evaluation value calculated by the processing unit PU is SAD in units of 4 × 4 pixels. In order to support both frame images and field images, four adders 12 corresponding to frame images and four adders 13 corresponding to field images are provided. The adder 12 calculates and outputs the SAD of four consecutive rows (16 in total) of the computing elements PE. The adder 13 calculates and outputs SADs of four rows (16 rows in total) of computing elements PE in every other row (one skip).

  Here, when the pixel value thinned by 1/2 in the y direction (vertical direction) is input from the SWRAM 2 to the array 5, Frame_4 × 4_SAD is the SAD (or the SAD of the field image without thinning in the y direction) (or field_4 × 4_SAD corresponds to the SAD of the field image thinned by 1/2 in the y direction.

  When performing vertical direction DS (straight line search), Field_4 × 4_SAD is calculated according to the rotation of the input image. Therefore, a sub search using Field_4 × 4_SAD is possible even in the vertical direction DS.

  The processing unit PU has a data buffer of 8 × 8 pixel size (“register” in FIG. 7). This buffer can hold the pixel value of the reference image SW and the pixel value of the encoding target image TB in an 8 × 8 size. As an initial load, the buffer for the reference image SW is supplied with a pixel value of 8 × 8 pixels from the SWRAM 2 and can hold it in one cycle. Further, the processing unit PU can shift the pixel value of the reference image SW held therein by one pixel left and right. The pixel value overflowed by this shift can be supplied to the shift register unit SRU in the same sub-block SBSA or the processing unit PU in the laterally adjacent sub-block SBSA. Setting that the evaluation value calculation unit is not used (that is, the output from the calculation element PE is discarded) is also possible. In this case, the processing unit PU operates as an 8 × 8 size shift register. Furthermore, the processing unit PU can receive the pixel value supplied from the lower direction and shift the pixel value held therein by one pixel in the vertical direction. The pixel value overflowed by this shift can be supplied to the processing unit PU in the upper adjacent sub-block SBSA.

<Shift register unit SRU>
The shift register unit SRU is a pixel value buffer for a total of 64 pixels of 8 × 8 pixels, and includes 8 × 8 shift register elements SRE. FIG. 8 shows an internal configuration of the shift register element SRE alone. As an initial load, the shift register unit SRU receives a pixel value of 8 × 8 pixels from the SWRAM 2 and can hold it in one cycle. The shift register unit SRU can shift the pixel value held inside by one pixel to the left and right, and the pixel value overflowed by this shift is the processing unit PU in the same sub-block SBSA or the horizontal adjacent sub-block. It is possible to supply to the shift register unit SRU in the SBSA. Further, the shift register unit SRU can receive the pixel value supplied from the lower direction and shift the held pixel value by one pixel in the vertical direction. The pixel value overflowed by this shift can be supplied to the shift register unit SRU in the upper adjacent sub-block SBSA.

  Referring to FIG. 4, storage unit (REG_VS) 7 is a data buffer register for realizing a vertical search during an FS (Full Search) operation. The internal configuration of the storage unit 7 is shown in FIG. As an external input to the storage unit 7, there is an 8 × 1 pixel value input from the SWRAM 2. Further, as external inputs from the storage unit 7, an 8 × 1 pixel value output to the processing unit PU of each sub-block SBSA and an 8 × 1 pixel output to the shift register unit SRU of each sub-block SBSA There is a pixel value output. The storage unit 7 includes 16 pixel value buffers 15 in units of 8 × 1 pixels (eight for the processing unit PU and eight for the shift register unit SRU). Pixel value data input from the SWRAM 2 is input to one of the pixel value buffers 15 and held. The storage unit 7 can simultaneously output all the pixel value data held in the pixel value buffer 15 to the corresponding sub-block SBSA (processing unit PU and shift register unit SRU).

  Next, the operation of the image processing apparatus according to this embodiment will be described.

  Below, the whole operation | movement in each search method which can be implement | achieved is demonstrated. Search methods that can be executed by the image processing apparatus according to the present embodiment include (1) FS (Full Search), (2) DS (Directional Search), and (3) RBM (Random Block Matching). In the RRSA configuration, the configuration is changed not only by the search method but also by the block size for executing the search. Supported block sizes include 16 × 32, 16 × 16, 16 × 8, 8 × 16, and 8 × 8.

  An initial load, a horizontal shift operation, and a vertical shift operation are defined as basic operations of the sub-block SBSA. In the RRSA configuration, various search methods are realized by the same architecture by appropriately combining these basic operations according to the search method.

<Initial load>
Details of the pixel values that the processing unit PU, the shift register unit SRU, and the storage unit 7 hold in the initial load are shown in FIGS. FIG. 10 shows the case where the image is not rotated, and FIG. 11 shows the case where the image is rotated.

  Referring to FIG. 10, pixel values (a to h in FIG. 10) of an 8 × 8 image portion are loaded into processing unit PU in one cycle. The shift register unit SRU is loaded with the pixel values (“0” to “7” in FIG. 10) of the 8 × 8 image portion right adjacent to the image portion to be loaded into the processing unit PU in one cycle. The pixel value (“u” in FIG. 10) of the 8 × 1 image portion that is adjacent to the lower portion of the image portion loaded in the processing unit PU is loaded into the storage unit 7 in one cycle. In addition, the pixel value (“v” in FIG. 10) of the 8 × 1 image portion that is adjacent to the lower portion of the image portion loaded in the shift register unit SRU is loaded into the storage unit 7 for one cycle.

  Referring to FIG. 9, since it is impossible to load pixel values to all 16 pixel value buffers 15 in the storage unit 7 at the initial load at the same time, the pixel values are sequentially loaded for each pixel value buffer 15. It is necessary to let them.

  Referring to FIG. 11, in the processing unit PU, the pixel values (a to h in FIG. 11) of the 8 × 8 image portion are rotated 90 ° counterclockwise by the image rotation processing unit 4 (see FIG. 1). However, it is loaded in one cycle. The shift register unit SRU stores the pixel values (“0” to “7” in FIG. 10) of the 8 × 8 image portion that is adjacent to the image portion loaded in the processing unit PU. Is loaded in one cycle while being rotated 90 ° counterclockwise.

<Left shift operation>
The left shift operation is an operation that shifts the pixel value in the left direction from the current holding state, and is a basic operation of the linear continuous point search. The search is an operation of performing a continuous point search from left to right. FIG. 12 shows the left shift operation. The pixel values (“0” in FIG. 12) for the 8 pixels in the left end column of the shift register unit SRU are supplied to the processing unit PU and held in the 8 pixels in the right end column of the processing unit PU. The pixel value overflowing from the processing unit PU (“a” in FIG. 12) can be held in 8 pixels in the right end column of the shift register unit SRU.

<Right shift operation>
The right shift operation is an operation for shifting the pixel value in the right direction from the current holding state, and is an operation necessary for realizing the snake search (see FIG. 15) as the FS operation. As the search, a continuous point search is performed from right to left. The right shift operation is shown in FIG. Pixel values corresponding to 8 pixels in the right end column of the processing unit PU ("h" in FIG. 13) are supplied to the shift register unit SRU and held in the 8 pixels in the left end column of the shift register unit SRU. The pixel value overflowing from the shift register unit SRU (“7” in FIG. 13) can be held in the leftmost column of 8 pixels of the processing unit PU.

<Upshift operation>
The upward shift operation is an operation for shifting the pixel value upward from the current holding state, and is an operation necessary for realizing a snake search (see FIG. 15) as the FS operation. The search is an operation of shifting by one pixel from top to bottom. FIG. 14 shows the upshift operation. The pixel value of 8 × 1 pixels (“u” in FIG. 14) held in the storage unit 7 (or the processing unit PU in the lower adjacent sub-block SBSA) is supplied to the processing unit PU, and the processing unit PU It is held at the bottom row of 8 pixels. The pixel values for the upper 8 rows of pixels overflowing from the processing unit PU are held in the lower 8 rows of the processing unit PU in the upper adjacent sub-block SBSA or discarded.

  Further, the 8 × 1 pixel value (“v” in FIG. 14) held in the storage unit 7 (or the shift register unit SRU in the lower adjacent sub-block SBSA) is supplied to the shift register unit SRU, It is held in the bottom row of 8 pixels of the shift register unit SRU. The pixel values for the upper 8 rows of pixels overflowing from the shift register unit SRU are held in the lower row 8 pixels of the shift register unit SRU in the upper adjacent sub-block SBSA or discarded.

<FS (Full Search)>
FS is a general search method, and is a method for exhaustively searching a rectangular area designated as a search range. In the RRSA configuration, FS is realized by a method called snake search. The search order in the rectangular area in the snake search is shown in FIG. Moreover, the internal connection state in the sub-block SBSA at the time of FS is shown in FIGS. 16 corresponds to a macroblock pair (16 × 32), FIG. 17 corresponds to mode 1 (16 × 16), FIG. 18 corresponds to mode 2 (16 × 8), and FIG. 19 corresponds to mode 3 (8 × 16). FIG. 20 corresponds to mode 4 (8 × 8).

  When the block size is a macroblock pair, mode 1 and mode 2, as shown in FIGS. 16 to 18, a connection path is formed between horizontally adjacent sub-blocks SBSA, while when the mode size is mode 3 and mode 4, FIG. As shown at 20, no connection path is formed between the horizontally adjacent sub-blocks SBSA.

  The input from the lower direction is also determined according to the block size. When the block size is a macroblock pair, mode1, and mode3, as shown in FIGS. 16, 17, and 19, the connection path is between vertical adjacent subblocks SBSA. On the other hand, in the case of mode 2 and mode 4, a connection path is formed with the storage unit 7 as shown in FIGS. However, also in FIGS. 16, 17, and 19, the sub-block SBSA located at the bottom forms a connection path with the storage unit 7.

  As shown in FIG. 15, in the snake search, the operation is repeated in the order of left shift → upshift → right shift → upshift → left shift →. The pixel values held by the processing unit PU and the shift register unit SRU are used to the maximum, the maximum horizontal search range is ± 4 or less for mode 3 and mode 4, and in the case of a macroblock pair, mode 1 and mode 2 If it is ± 8 or less, neither an intermediate load is required for both the processing unit PU and the shift register unit SRU in the FS. However, in order to complete the FS without causing a stall, the pixel value of the next row is loaded into the storage unit 7 between the time when the upper shift is executed and the time when the next upper shift is executed. There is a need.

  Further, in the image processing apparatus according to the present embodiment, two processing units PU and two shift register units SRU are connected in series in the horizontal direction using the connection between the horizontally adjacent sub-blocks SBSA. It is possible. Accordingly, when the required parallelism is 256 or less (that is, 4 or less sub-blocks SBSA are used) in mode 3 or mode 4, one processing unit PU is used by using one processing unit PU as a shift register. And three shift register units SRU can be used as a serial connection. In this case, the maximum search range without an intermediate load can be expanded to ± 12.

<DS (Directional Search)>
DS is a search method that performs a straight line search horizontally or vertically. The internal connection state in the sub-block SBSA at the time of DS is shown in FIGS. 21 corresponds to a macroblock pair (16 × 32), FIG. 22 corresponds to mode 1 (16 × 16), FIG. 23 corresponds to mode 2 (16 × 8), and FIG. 24 corresponds to mode 3 (8 × 16). FIG. 25 corresponds to mode 4 (8 × 8).

  In the case of the vertical search, the pixel value input to the data path from the SWRAM 2 is rotated 90 ° counterclockwise by the image rotation processing unit 4 (see FIG. 1). Since the size of the array 5 is 16 × 32 pixels and the rotated macroblock pair cannot be held in the array 5, the vertical search is not possible for the macroblock pair. However, by expanding the horizontal size of the array 5 to 32 pixels or more, it is possible to perform a vertical search for a macroblock pair.

  The DS search is performed using only the left shift (FIG. 12). When the block size is a macroblock pair, mode 1 and mode 2, as shown in FIGS. 21 to 23, a connection path is formed between horizontally adjacent sub-blocks SBSA, while when the mode size is mode 3 and mode 4, FIG. As shown in FIG. 25, no connection path is formed between the horizontally adjacent sub-blocks SBSA.

  With respect to the shift register unit SRU, it is necessary to perform one intermediate load every time eight points are searched. At the time of intermediate loading, pixel values for 8 × 8 pixels are supplied from the SWRAM 2 to the shift register unit SRU in one cycle.

<RBM (Random Block Matching>
RBM is a search method for searching only a single point. With respect to the search for a single point, it is not necessary to perform a shift operation, and the evaluation value at that point is automatically obtained by simply performing an initial load on the processing unit PU.

  In FS, DS, and RBM, when the block size is a macroblock pair, eight sub-blocks SBSA are connected to form one 16 × 32 block. In this case, as shown in FIG. 26A, only one block can be configured in the array 5.

  Similarly, in the case of mode 1, four (2 horizontal × 2 vertical) sub-blocks SBSA are connected to form a 16 × 16 block. In this case, as shown in FIG. 26B, a maximum of two blocks can be configured in the array 5. Each of the two blocks can operate independently of the other blocks.

  Similarly, in the case of mode 2, two horizontal sub-blocks SBSA are connected to form a 16 × 8 block. In this case, as shown in FIG. 26C, a maximum of four blocks can be configured in the array 5. Each of the four blocks can operate independently of the other blocks.

  Similarly, in the case of mode 3, two vertical sub-blocks SBSA are connected to form an 8 × 16 block. In this case, as shown in FIG. 26D, a maximum of four blocks can be configured in the array 5. Each of the four blocks can operate independently of the other blocks.

  Similarly, in the case of mode 4, an 8 × 8 block is configured by one sub-block SBSA. In this case, as shown in FIG. 27E, a maximum of 8 blocks (equivalent to sub-blocks) can be configured in the array 5. Each of the eight blocks can operate independently of the other blocks.

  In addition, as shown in FIGS. 27F and 27G, two types of modes can be executed simultaneously. Further, as shown in FIG. 27H, three types of modes can be executed simultaneously. Even in such a case, each of the plurality of blocks can operate independently of the other blocks. Further, similarly to the simultaneous use of a plurality of types of block sizes, a plurality of types of search methods can be executed simultaneously. For example, in FIG. 26B, it is possible to perform the FS operation in the upper 16 × 16 block and simultaneously perform the DS operation in the lower 16 × 16 block.

<Summary>
Thus, according to the image processing apparatus according to the present embodiment, array 5 is divided into a plurality of sub-blocks SBSA0 to SBSA7. Then, by switching the settings of the multiplexers 10A, 10B, 11A, and 11B according to the size of the image to be processed, one or a plurality of blocks including one or a plurality of sub-blocks SBSA are set in the array 5. For this reason, even if the number of blocks is increased by subdividing the macro block pair to be processed, a plurality of blocks set in the array 5 can be processed at the same time, so the number of cycles required for searching for one macro block pair is reduced. The increase can be avoided. Further, since each of the plurality of blocks can operate independently of the other blocks, a plurality of blocks in one macroblock pair can be processed in parallel. As a result, an increase in the number of cycles for searching for one macroblock pair can be avoided.

  The result of having verified the effect of this invention by numerical calculation is shown to FIGS. Regarding the number of cycles and the amount of data transferred from the SRAM, the RRSA configuration according to the present invention is compared with the conventional SIMD configuration and RCSA configuration.

  For each mode, normalization is performed with the number of cycles and the data transfer amount set to 100% in the SIMD configuration. FIG. 28 is an estimate when the FS operation is performed in the search range ± 4 × ± 4 (91 points in total). FIG. 29 is an estimate when the FS operation is performed with a search range of ± 8 × ± 8 (total of 289 points). FIG. 30 is an estimate when the DS operation is performed with 33 consecutive search ranges. FIG. 31 shows an estimate when the RBM operation is performed with a search range random of 16 points. Referring to FIGS. 28 to 31, in all cases, it can be seen that the RRSA configuration according to the present invention can reduce the number of cycles and the amount of data transfer compared to the conventional SIMD configuration and RCSA configuration.

1 is a diagram illustrating an overall configuration of an image processing apparatus according to the present invention. It is a figure which shows the image block which can be read from SWRAM. It is a figure which shows arrangement | positioning of the pixel input into an array when an image is rotated. It is a figure which shows the whole structure of an array. It is a figure which shows the internal structure of a subblock. It is a figure which shows the internal structure of the evaluation value calculation part in a processing unit. It is a figure which shows the internal structure of a calculation element single-piece | unit. It is a figure which shows the internal structure of a shift register element single-piece | unit. It is a figure which shows the internal structure of a memory | storage part. It is a figure which shows the detail about the pixel value which a processing unit, a shift register unit, and a memory | storage part hold | maintain by initial load. It is a figure which shows the detail about the pixel value which a processing unit, a shift register unit, and a memory | storage part hold | maintain by initial load. It is a figure which shows left shift operation | movement. It is a figure which shows a right shift operation | movement. It is a figure which shows an up shift operation | movement. It is a figure which shows the search order in the rectangular area in a snake search. It is a figure which shows the internal connection state in the subblock at the time of FS. It is a figure which shows the internal connection state in the subblock at the time of FS. It is a figure which shows the internal connection state in the subblock at the time of FS. It is a figure which shows the internal connection state in the subblock at the time of FS. It is a figure which shows the internal connection state in the subblock at the time of FS. It is a figure which shows the internal connection state in the subblock at the time of DS. It is a figure which shows the internal connection state in the subblock at the time of DS. It is a figure which shows the internal connection state in the subblock at the time of DS. It is a figure which shows the internal connection state in the subblock at the time of DS. It is a figure which shows the internal connection state in the subblock at the time of DS. It is a figure which shows the setting of the block in an array. It is a figure which shows the setting of the block in an array. It is a figure which shows the result of having verified the effect of this invention. It is a figure which shows the result of having verified the effect of this invention. It is a figure which shows the result of having verified the effect of this invention. It is a figure which shows the result of having verified the effect of this invention. It is a figure which shows the example of a SIMD structure. It is a figure which shows the example of a RCSA structure. It is a figure which shows the internal structure of an arithmetic element. It is a figure which shows the internal structure of a shift register.

Explanation of symbols

1 IME core 2 SWRAM
3 TB buffer 4 Image rotation processing unit 5 Array 7 Storage unit 10A, 10B, 11A, 11B Multiplexer

Claims (7)

A plurality of computing elements for computing evaluation values based on pixel values of an image are provided in an array,
The array is divided into a plurality of sub-blocks each including a predetermined number of the arithmetic elements,
Each of the plurality of sub-blocks has selection means capable of selecting whether or not to connect the own sub-block and an adjacent sub-block adjacent to the own sub-block,
An image processing apparatus capable of setting one or a plurality of blocks including one or a plurality of sub-blocks in the array by switching the setting of the selection unit according to the size of an image to be processed.   The image processing apparatus according to claim 1, wherein when a plurality of blocks are set in the array, each of the plurality of blocks can operate independently of the other blocks. The sub-block is
A first unit having a plurality of arithmetic elements;
A second unit having a plurality of registers that can be calculated by the calculation element in the first unit or that can hold the calculated pixel value;
The selection means includes
Selection means for selecting one of the second unit of the own subblock and the first unit of the adjacent subblock as an input to the first unit of the own subblock;
The image processing apparatus according to claim 1, further comprising selection means for selecting one of the first unit of the own subblock and the second unit of the adjacent subblock as an input to the second unit of the own subblock.   When a plurality of first units and a plurality of second units are included in one block by connecting a plurality of the sub-blocks, some first units of the plurality of first units The image processing apparatus according to claim 3, wherein the image processing apparatus can be used as a register for calculating or calculating a pixel value calculated by an arithmetic element in another first unit. A storage unit capable of holding a pixel value of an image portion at a location adjacent to the image portion loaded in the array in a predetermined direction;
The image processing according to claim 1, wherein when the evaluation position of the image is shifted in the predetermined direction, a pixel value held in the storage unit is input from the storage unit to the array. apparatus. The selection means includes
The image processing apparatus according to claim 5, further comprising selection means for selecting one of an adjacent sub-block and the storage unit as an input to the own sub-block. The sub-block has an adder group that adds evaluation values calculated by the plurality of arithmetic elements in the sub-block.
The adder group is:
An adder group corresponding to frame images for adding evaluation values of consecutive rows;
The image processing apparatus according to claim 1, further comprising an adder group corresponding to a field image for adding an evaluation value for every other row.

Images