JP5296656B2 - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To solve the problem that a higher resolution image is large in data size and its update is not easy. <P>SOLUTION: A graphic processor 402 performs rendering by using model data 418 stored in a main memory 60 according to an image processing request from a main processor 400 and updates a color buffer 414 and Z buffer 416. Both the color buffer 414 and the Z buffer 416 have configurations of hierarchical data comprising a plurality of pixel planes corresponding to a plurality of resolutions of an image to be drawn. According to the request from the main processor 400, a rendering part 404 of the graphic processor 402 performs rendering and a hierarchical data update part 406 performs scaling processing of a rendering result and reflects it on each hierarchy of the color buffer 414 and the Z buffer 416. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image processing technique for enlarging / reducing an image displayed on a display, or moving the image vertically and horizontally.

  Home entertainment systems have been proposed that not only execute game programs, but also play video. In this home entertainment system, the GPU generates a three-dimensional image using polygons (see, for example, Patent Document 1).

  Regardless of the purpose of image display, how efficiently an image is displayed is always an important issue. In particular, various ideas are required to draw a high-definition image at high speed. For example, a technique for efficiently storing texture data and performing mapping efficiently has been proposed (see, for example, Non-Patent Documents 1 and 2). .

US Pat. No. 6,563,999

Sylvain Fefebvre, et.al., Unified Texture Management for Arbitrary Meshes, Repport de recherche, N5210, May 2004, Institut National De Recherche En Informatique Et En Automatique Martin Kraus, et.al., Adaptive Texture Maps, Graphics Hardware (2002), pp1-10, The Eurographics Association

  Even when an image has a high definition, keeping the data size small and drawing at high speed are always important issues for displaying the image with high responsiveness. In addition, when it becomes necessary to correct a part of an image, even if the technology for devising the data structure such as the texture mapping described above is applied, it is necessary to update all the data structures once constructed. There is a problem that is likely to become complicated.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an image processing technique that can efficiently generate, display, and correct data size even when the image size is large. There is.

  One embodiment of the present invention relates to an image processing apparatus. The image processing apparatus includes an image processing request unit that issues an image processing request including information for drawing an image to be newly displayed by executing a program, and a color that holds color information for each pixel of an image to be drawn Hierarchical data storage that stores a hierarchical color buffer and a hierarchical Z buffer in which a value pixel plane and a Z value pixel plane that holds depth information from the viewpoint for each pixel are hierarchized in correspondence with a plurality of resolutions of the drawing target image And an image processing unit that accepts an image processing request, executes image processing, and updates the hierarchical color buffer and the hierarchical Z buffer. The image processing unit reads the drawing target read from the hierarchical color buffer and the hierarchical Z buffer. A drawing color buffer and a drawing Z buffer for recording data of each layer and area, and a drawing target hierarchy and A drawing unit that specifies an area, reads data into a drawing color buffer and a drawing Z buffer, performs image processing calculation based on an image processing request and updates the data, and the updated drawing color buffer and drawing Z buffer And a hierarchical data updating unit that performs scaling processing on data in accordance with a plurality of resolutions and updates data of each layer of the hierarchical color buffer and the hierarchical Z buffer.

  Yet another embodiment of the present invention relates to an image processing method. This image processing method includes a step of issuing an image processing request including information for drawing an image to be newly displayed by executing a program, and a color value pixel plane that holds color information for each pixel of an image to be drawn Storing a hierarchical color buffer and a hierarchical Z buffer in which a Z-value pixel plane holding depth information from the viewpoint for each pixel is hierarchized in correspondence with a plurality of resolutions of the drawing target image in a memory; Receiving a processing request, executing image processing, and updating the hierarchical color buffer and the hierarchical Z buffer, wherein the step of updating the hierarchical color buffer and the hierarchical Z buffer includes: The data is read from the hierarchical color buffer and hierarchical Z buffer, and the drawing color buffer And writing to the drawing Z buffer, updating the drawing color buffer and drawing Z buffer data by performing image processing calculation based on the image processing request, and updating the drawing color buffer and drawing Z buffer. Performing a scaling process on the data in correspondence with a plurality of resolutions, and updating data of each layer of the layer color buffer and layer Z buffer.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to provide an image processing apparatus capable of efficiently constructing image data for displaying an image with a wide range of resolutions.

It is a figure which shows the use environment of the information processing system concerning the form of Embodiment 1. It is a figure which shows the external appearance structure of the input device applicable to the image processing system of FIG. 3 is a conceptual diagram of a hierarchical structure of image data used in Embodiment 1. FIG. 1 is a diagram illustrating a configuration of an information processing device in Embodiment 1. FIG. 3 schematically shows a flow of image data in the first embodiment. 6 is a diagram for explaining prefetching processing of image data in the first embodiment. FIG. 3 is a diagram illustrating in detail a configuration of a control unit having a function of displaying hierarchical data in the first embodiment. FIG. 3 is a diagram schematically illustrating a data structure of hierarchical data used in Embodiment 1. FIG. 3 is a diagram illustrating a configuration of a control unit having a function of generating a display target image file in Embodiment 1. FIG. 6 is a diagram for explaining a relationship between an original image and a tile image when there is redundancy in the image in Embodiment 1. FIG. FIG. 10 is a diagram for explaining another example of the relationship between an original image and a tile image when there is redundancy in the image in the first embodiment. FIG. 10 is a diagram for describing a technique for defining sharing of tile images based on redundancy by a header in the first embodiment. 6 is a diagram for describing a technique for defining sharing of tile images based on redundancy by index blocks in Embodiment 1. FIG. It is a flowchart which shows the process sequence which the control part shown in FIG. 9 produces | generates an image file. 4 is a flowchart illustrating a processing procedure for the control unit to display an image file in the first embodiment. FIG. 2 is a diagram illustrating a configuration of a control unit having a function of correcting an image in the first embodiment. 6 is a diagram illustrating an example of correcting an image in the first embodiment. FIG. It is a figure which shows typically the change of the pointer of the index block in the correction like FIG. FIG. 10 is a diagram for describing processing when a hierarchy is added to the high resolution side in the first embodiment. 6 is a diagram for explaining changes in headers and index blocks when there is no index block to which a hierarchy to be added belongs in Embodiment 1. FIG. FIG. 10 is a diagram for explaining processing for adding a new area to an existing image in the first embodiment. 6 is a diagram for explaining changes in headers and index blocks when a new area is added in Embodiment 1. FIG. FIG. 10 is a diagram for describing a method of dividing an area defined in a header when a hierarchy is added to the vertex side of a hierarchical structure in the first embodiment. 4 is a flowchart illustrating a processing procedure when a user modifies and modifies an image in the image processing apparatus according to the first embodiment. It is a figure which shows the structure of a control part and a main memory among the information processing apparatuses in Embodiment 2 in detail. 10 is a flowchart illustrating a processing procedure of rendering to hierarchical data in the second embodiment. FIG. 10 is a diagram illustrating a device configuration when a scaler that performs scaling processing is built in a main memory in the second embodiment. FIG. 10 is a diagram illustrating a state of addressing of a GPU buffer in the second embodiment. FIG. 10 is a diagram illustrating a state of addressing of a GPU buffer in the second embodiment.

Embodiment 1
The image data to be processed in the present embodiment has a hierarchical structure composed of images with different resolutions generated by reducing the original image in a plurality of stages. Each layer image is divided into one or a plurality of tile images. For example, the image with the lowest resolution is composed of one tile image, and the original image with the highest resolution is composed of the largest number of tile images. At the time of image display, the tile image used for drawing is switched to a tile image of a different hierarchy when the display image has a predetermined resolution, so that enlarged display or reduced display is quickly performed.

  First, a basic display mode of an image having such a hierarchical structure will be described. FIG. 1 shows a use environment of an information processing system 1 to which an embodiment of the present invention can be applied. The information processing system 1 includes an information processing device 10 that executes an application program including image processing, and a display device 12 that outputs a processing result by the information processing device 10. The display device 12 may be a television having a display that outputs an image and a speaker that outputs sound.

  The display device 12 may be connected to the information processing device 10 by a wired cable, or may be wirelessly connected by a wireless local area network (LAN) or the like. In the information processing system 1, the information processing apparatus 10 may be connected to an external network such as the Internet via the cable 14 to download and acquire content including hierarchical compressed image data. Note that the information processing apparatus 10 may be connected to an external network by wireless communication.

  In response to a request from the user, the information processing apparatus 10 performs a process of changing the display area, such as an enlargement / reduction process of an image displayed on the display of the display device 12 or a movement process in the vertical and horizontal directions. When the user operates the input device while viewing the image displayed on the display, the input device transmits a display area change request signal to the information processing device 10.

  FIG. 2 shows an external configuration of the input device 20. The input device 20 includes a cross key 21, analog sticks 27a and 27b, and four types of operation buttons 26 as operation means that can be operated by the user. The four types of operation buttons 26 include a circle button 22, a x button 23, a square button 24, and a triangle button 25.

  In the information processing system 1, a function for inputting a display image enlargement / reduction request and a vertical / left / right scroll request is assigned to the operation unit of the input device 20. For example, the input function of the display image enlargement / reduction request is assigned to the right analog stick 27b. The user can input a display image reduction request by pulling the analog stick 27b forward, and can input a display image enlargement request by pressing the analog stick 27b from the front. The input function of the display area movement request is assigned to the cross key 21. The user can input a movement request in the direction in which the cross key 21 is pressed by pressing the cross key 21. Note that the image change request input function may be assigned to another operation means, for example, the scroll request input function may be assigned to the analog stick 27a.

  In addition, in order to realize various functions described later, the input device 20 is also assigned a function of moving a cursor displayed on the image and selecting a file or a command. Alternatively, the input device 20 may be realized by a general input device such as a pointing device, a mouse, a keyboard, or a touch panel. The function assignment as described above may be appropriately determined according to the type of the input device 20.

  The input device 20 has a function of transmitting an input display area change request signal or the like to the information processing device 10, and is configured to be capable of wireless communication with the information processing device 10 in the present embodiment. The input device 20 and the information processing device 10 may establish a wireless connection using a Bluetooth (registered trademark) protocol, an IEEE802.11 protocol, or the like. The input device 20 may be connected to the information processing apparatus 10 via a cable and transmit a display area change request signal or the like to the information processing apparatus 10.

  FIG. 3 is a conceptual diagram of a hierarchical structure of image data used in the present embodiment. The image data has a hierarchical structure including a 0th hierarchy 30, a first hierarchy 32, a second hierarchy 34, and a third hierarchy 36 in the depth (Z-axis) direction. Although only four layers are shown in the figure, the number of layers is not limited to this. Hereinafter, image data having such a hierarchical structure is referred to as “hierarchical data”. However, the hierarchical data in the figure is conceptual, and actually the hierarchical data is expressed by a plurality of data sets as described later.

  The hierarchical data shown in FIG. 3 has a hierarchical structure of a quadtree, and each hierarchy is composed of one or more tile images 38. All the tile images 38 are formed in the same size having the same number of pixels, and have, for example, 256 × 256 pixels. The image data of each layer expresses one image at different resolutions, and the original image of the third layer 36 having the highest resolution is reduced in a plurality of stages to obtain the second layer 34, the first layer 32, the 0th layer. Image data of the hierarchy 30 is generated. For example, the resolution of the Nth layer (N is an integer greater than or equal to 0) may be ½ of the resolution of the (N + 1) th layer in both the left and right (X axis) direction and the up and down (Y axis) direction.

  In the information processing apparatus 10, the hierarchical data is stored in the storage device in a compressed state in a predetermined compression format, and is read from the storage device and decoded before being displayed on the display. The information processing apparatus 10 according to the present embodiment has a decoding function corresponding to a plurality of types of compression formats, and can decode compressed data in, for example, the S3TC format, JPEG format, and JPEG2000 format.

  As shown in FIG. 3, the hierarchical structure of the hierarchical data is set with the horizontal direction as the X axis, the vertical direction as the Y axis, and the depth direction as the Z axis, thereby constructing a virtual three-dimensional space. When the information processing device 10 derives the change amount of the display image from the display area change request signal supplied from the input device 20, the information processing device 10 derives the coordinates (frame coordinates) of the four corners of the frame in the virtual space using the change amount. . The frame coordinates in the virtual space are used for loading compressed data into a main memory, which will be described later, and display image generation processing. Instead of the frame coordinates in the virtual space, the information processing apparatus 10 may derive information for specifying a hierarchy and texture coordinates (UV coordinates) in the hierarchy. Hereinafter, the combination of the hierarchy specifying information and the texture coordinates is also referred to as frame coordinates.

  FIG. 4 shows the configuration of the information processing apparatus 10. The information processing apparatus 10 includes a wireless interface 40, a switch 42, a display processing unit 44, a hard disk drive 50, a recording medium mounting unit 52, a disk drive 54, a main memory 60, a buffer memory 70, and a control unit 100. . The display processing unit 44 has a frame memory that buffers data to be displayed on the display of the display device 12.

  The switch 42 is an Ethernet switch (Ethernet is a registered trademark), and is a device that transmits and receives data by connecting to an external device in a wired or wireless manner. The switch 42 is connected to an external network via the cable 14 and configured to receive content data and the like from the server. The switch 42 is connected to the wireless interface 40, and the wireless interface 40 is connected to the input device 20 using a predetermined wireless communication protocol. A signal input from the user in the input device 20 is supplied to the control unit 100 via the wireless interface 40 and the switch 42.

  The hard disk drive 50 functions as a storage device that stores data. Hierarchical data received via the switch 42 is stored in the hard disk drive 50. When a removable recording medium such as a memory card is mounted, the recording medium mounting unit 52 reads data from the removable recording medium. When a read-only ROM disk is loaded, the disk drive 54 drives and recognizes the ROM disk to read data. The ROM disk may be an optical disk or a magneto-optical disk. Hierarchical data may be stored in these recording media.

  The control unit 100 includes a multi-core CPU, and includes one general-purpose processor core and a plurality of simple processor cores in one CPU. The general-purpose processor core is called PPU (PowerPC Processor Unit), and the remaining processor cores are called SPU (Synergistic Processor Unit).

  The control unit 100 includes a memory controller connected to the main memory 60 and the buffer memory 70. The PPU has a register, has a main processor as an operation execution subject, and efficiently assigns a task as a basic processing unit in an application to be executed to each SPU. Note that the PPU itself may execute the task. The SPU has a register, and includes a sub-processor as an operation execution subject and a local memory as a local storage area. The local memory may be used as the buffer memory 70.

  The main memory 60 and the buffer memory 70 are storage devices and are configured as a RAM (Random Access Memory). The SPU has a dedicated DMA (Direct Memory Access) controller as a control unit, can transfer data between the main memory 60 and the buffer memory 70 at high speed, and the frame memory and the buffer memory 70 in the display processing unit 44 can be transferred. High-speed data transfer can be realized. The control unit 100 according to the present embodiment realizes a high-speed image processing function by operating a plurality of SPUs in parallel. The display processing unit 44 is connected to the display device 12 and outputs an image processing result according to a request from the user.

  The information processing apparatus 10 according to the present embodiment uses a main part of the compressed image data from the hard disk drive 50 in order to smoothly change the display image when the display image enlargement / reduction process or the display area movement process is performed. It is loaded into the memory 60. Further, a part of the compressed image data loaded in the main memory 60 is decoded and stored in the buffer memory 70. This makes it possible to instantaneously switch an image used for generating a display image at a necessary later timing.

  FIG. 5 schematically shows the flow of image data in the present embodiment. First, the hierarchical data is stored in the hard disk drive 50. Instead of the hard disk drive 50, a recording medium mounted on the recording medium mounting unit 52 or the disk drive 54 may be held. Alternatively, the hierarchical data may be downloaded from an image server connected by the information processing apparatus 10 via a network. As described above, the hierarchical data here is subjected to fixed length compression in the S3TC format or the like, or variable length compression in the JPEG format or the like.

  Among the hierarchical data, a part of the image data is loaded into the main memory 60 in a compressed state (S10). The area to be loaded here is a predetermined rule such as the vicinity of the current display image in the virtual space, the area of the image, the area where a display request is predicted to be frequently made based on the user's browsing history, etc. Determined by. The loading is performed not only when an image change request is made, but also at any given time interval, for example. This prevents the load process from being concentrated at one time.

  Next, among the compressed image data stored in the main memory 60, the tile image data of the area necessary for display or the tile image data of the area predicted to be necessary is decoded and stored in the buffer memory 70 (S12). ). The buffer memory 70 includes at least two buffer areas 72 and 74. The size of each of the buffer areas 72 and 74 is set to be larger than the size of the frame memory 90. When a display area change request signal is input from the input device 20, the buffer area A display image can be generated with the image data expanded into 72 and 74.

  One of the buffer areas 72 and 74 is a display buffer for holding an image used for generating a display image, and the other is a decoding buffer for preparing an image predicted to be necessary thereafter. In the example of FIG. 5, it is assumed that the buffer area 72 is a display buffer, the buffer area 74 is a decoding buffer, and the display area 68 is displayed. The image stored in the decoding buffer by the prefetching process to be described later may be an image in the same hierarchy as the image stored in the display buffer, or may be an image in a different hierarchy having a different scale.

  Next, the image in the display area 68 among the images stored in the buffer area 72, which is a display buffer, is drawn in the frame memory 90 (S14). During this time, the image in the new area is decoded as necessary and stored in the buffer area 74. The display buffer and the decoding buffer are switched according to the timing of completion of storage, the change amount of the display area 68, and the like (S16). Thereby, a display image can be smoothly switched with respect to a movement of a display area, a change of a scale ratio, or the like.

  FIG. 6 is a diagram for explaining the prefetching process. FIG. 6 shows the structure of hierarchical data, and each hierarchy is expressed as L0 (0th hierarchy), L1 (1st hierarchy), L2 (2nd hierarchy), and L3 (3rd hierarchy). In the hierarchical data structure shown in FIG. 6, the position in the depth (Z-axis) direction indicates the resolution. The position closer to L0 has a lower resolution, and the position closer to L3 has a higher resolution. When attention is paid to the size of the image displayed on the display, the position in the depth direction corresponds to the scale ratio, and when the scale ratio of the display image of L3 is 1, the scale ratio in L2 is 1/4, and in L1 The scale factor is 1/16, and the scale factor at L0 is 1/64.

  Therefore, in the depth direction, when the display image changes from the L0 side to the L3 side, the display image enlarges. When the display image changes from the L3 side to the L0 side, the display image is reduced. Go. An arrow 80 indicates that the display area change request signal from the user requests reduction of the display image, and crosses the reduction ratio ¼ (L2). In the information processing apparatus 10, the position in the depth direction of L1 and L2 prepared as the tile image 38 is set as the prefetch boundary in the depth direction, and when the image change request signal crosses the prefetch boundary, the prefetch processing is started. To do.

  When the scale factor of the display image is in the vicinity of L2, the display image is created using a tile image of L2 (second layer). Specifically, the L2 tile image is used when the scale ratio of the image to be displayed is between the switching boundary 82 between the L1 tile image and the L2 tile image and the switching boundary 84 between the L2 tile image and the L3 tile image. The Therefore, when the image reduction processing is requested as indicated by the arrow 80, the L2 tile image is converted from the enlarged image to the reduced image and displayed. On the other hand, a future tile image 38 predicted from the image change request signal is identified and decoded. In the example of FIG. 6, when the requested scale ratio due to the display area change request signal crosses L2, the information processing apparatus 10 prefetches the corresponding tile image 38 of L1 in the reduction direction from the hard disk drive 50 or the main memory 60. Are decoded and written into the buffer memory 70.

  Although the prefetch processing in the depth direction has been described above, the prefetch processing in the up, down, left, and right directions is similarly processed. Specifically, a prefetch boundary is set for the image data developed in the buffer memory 70, and the prefetch process is started when the display position by the image change request signal crosses the prefetch boundary.

  FIG. 7 shows in detail the configuration of the control unit 100a having the function of displaying the hierarchical data described above in the present embodiment. The control unit 100a includes an input information acquisition unit 102 that acquires information input by the user from the input device 20, a tile image specification unit 110 that specifies a tile image including a region to be newly displayed, and image data to be newly loaded. A load block determining unit 106 for determining and a load unit 108 for loading necessary image blocks from the hard disk drive 50 are included. The control unit 100a further includes a decoding unit 112 that decodes the compressed image data and a display image processing unit 114 that draws a display image.

  In FIG. 7, each element described as a functional block for performing various processes can be configured by a CPU (Central Processing Unit), a memory, and other LSIs in terms of hardware. This is realized by a program loaded on the computer. As described above, the control unit 100 includes one PPU and a plurality of SPUs, and each functional block can be configured by the PPU and the SPU individually or in cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  The input information acquisition unit 102 acquires instruction contents such as start / end of image display, movement of a display area, enlargement and reduction of a display image, which are input by the user to the input device 20. The tile image specifying unit 110 specifies a tile image including an area to be newly displayed according to the frame coordinates of the current display area and the display area change request information input by the user. If the tile image is already loaded in the main memory 60, the information is supplied to the decoding unit 112, and if not loaded, the information is supplied to the load block determining unit 106. The tile image specifying unit 110 may specify not only an image necessary for drawing the display image at that time but also a tile image predicted to be necessary thereafter.

  Based on the information from the tile image specifying unit 110, the load block determining unit 106 specifies an image block composed of a plurality of tile images to be newly loaded from the hard disk drive 50 to the main memory 60, and requests the load unit 108 to load it. Is issued. The load block determination unit 106 may make a load request at a predetermined timing in a state where the load unit 108 is not performing the load process, for example, at a predetermined time interval or when the user makes a display area change request. The load unit 108 performs actual load processing in accordance with a request from the load block determination unit 106.

  The decoding unit 112 reads out and decodes the tile image data from the main memory 60 based on the tile image information acquired from the tile image specifying unit 110, and stores the decoded data in the decoding buffer or the display buffer. The display image processing unit 114 reads the corresponding image data from the display buffer of the buffer memory 70 based on the frame coordinates of the new display image, and draws it in the frame memory of the display processing unit 44.

  Next, an embodiment for constructing and outputting image data more efficiently in the image data display technology having the hierarchical structure described so far will be described. FIG. 8 schematically shows the data structure of hierarchical data in the present embodiment. In the present embodiment, the hierarchical data has a structure including three data of a header 150, an index block 160, and a tile image 170. The header 150 and the index block 160 are indexes for specifying the tile image to be processed based on the position in the virtual space formed by the hierarchical data. In the present embodiment, the hierarchical structure in the virtual space is divided into regions, and the indexes to the tile image data are grouped for each region, thereby improving the access to the tile images.

  The header 150 defines a plurality of areas obtained by dividing a pyramid-like hierarchical structure in the virtual space as shown in FIG. 3, and data in which a pointer to one of the index blocks 160 is set for each area. It is. In the header 150 shown in FIG. 8, a triangle 151 indicates the shape of the hierarchical structure viewed from the side, and horizontal broken lines indicate images 154a, 154b, 154c, 154d, 154e,. Such a hierarchical structure is divided into regions as indicated by solid lines, for example, and regions such as region 152a, region 152b, and region 152c are defined. In the header 150, in principle, a pointer to one of the index blocks 160 is set for each area.

  In the example of FIG. 8, the pointer to the index block 160a (arrow A) for the area 152a, the pointer to the index block 160b (arrow B) for the area 152b, and the pointer (arrow C) to the index block 160c for the area 152c. Shows a state where is set. However, as described later, a null pointer that does not point to a specific index block may be set for a part of the area defined by the header.

  The index block 160 is data that is generated for each region of the structure defined by the header 150, and a pointer to one of the tile images 170 is set to the position on the image of a plurality of layers belonging to each region. As shown in FIG. 8, one index block corresponding to one area in the header 150 includes data of at least a partial area of an image of a plurality of layers belonging to the area. In the example in the figure, the index block 160a corresponding to the area 152a holds the data of the entire two-layered images 154a and 154b belonging to the area 152a. The index block 160b corresponding to the area 152b holds the entire data of the three layers of images 154c, 154d, and 154e belonging to the area 152b.

  The index block 160c corresponding to the area 152c holds data of partial areas of the images 154f, 154g, and 154h for three layers belonging to the area 152c. For each tile area generated by dividing the image plane of each hierarchy into tile image sizes, a pointer indicating one of the tile images 170 that are actual image data of the tile area is held. However, as described later, when one tile area is drawn using a plurality of pieces of information, one tile area may hold pointers pointing to a plurality of tile images. In the figure, three planes respectively shown in the index blocks 160a, 160b, and 160 correspond to hierarchies belonging to the corresponding areas, and small rectangles on the respective hierarchies indicate tile areas.

  In the example of FIG. 8, in the index block 160b, a pointer (arrow D) to the tile image 170a is set for the tile area 164a, and a pointer (arrow E) to the tile image 170b is set for the tile area 164b. It shows a state. However, as described later, a part of the tile area defined by the index block may be a null pointer without setting a pointer to a specific tile image. The tile image 170 is compressed tile image data. By configuring the hierarchical data in this way, the tile image specifying unit 110 traces the header 150 and the index block 160 from the frame coordinates in the virtual space of the image frame to be displayed, and the tile image 170 included in the image frame. Can be specified.

  In FIG. 8, the index blocks 160a, 160b, and 160c are respectively provided with pointers to the index data for three layers of image areas composed of 4 × 4, 8 × 8, and 16 × 16 tile areas, that is, tile images 170. Shown to be configurable. As described above, when the number of tile areas that can be defined by one index block 160 is unified, for example, when any of the index blocks 160 is read from the hard disk drive 50 to the main memory 60, any index block is required. The size of the area does not change, and the storage area can be easily managed.

  Therefore, it is desirable that the area defined in the header 150 be generated so that the index block 160 has the same size. When the hierarchical data has a quadtree hierarchical structure in which the 0th hierarchy is one tile image as shown in FIG. 3, the 3rd hierarchy image belonging to the area 152b in FIG. 8, that is, the 2nd hierarchy image 154c, The third layer image 154d and the fourth layer image 154e are composed of tile areas of 4 × 4, 8 × 8, and 16 × 16, respectively.

  Therefore, when an area that can be defined by one index block 160 is an image area composed of 4 × 4, 8 × 8, and 16 × 16 tile areas as described above, the data of the entire three-layer image is directly used as one index. It can be stored in block 160b. That is, all the regions of the second layer image 154c, the third layer image 154d, and the fourth layer image 154e in the layer data are all of the 0th layer 162d, the first layer 162e, and the second layer 162f of the index block 160b. A pointer to the tile image can be set in correspondence with each region.

  On the other hand, the area 152a in the header 150 includes the second hierarchy images 154a and 154b of the 0th hierarchy and the 1st hierarchy, the 0th hierarchy image 154a is one tile area, and the 1st hierarchy image 154b is a 2 × 2 tile. Since it consists of areas, the number of layers and the number of tile areas are small compared to areas that can be defined by one index block 160a. In such a case, the 0th layer image 154a in the layer data is made to correspond to a part of the 0th layer 162a of the index block 160a, for example, one tile area at the upper left, and a pointer to the tile image is set. Similarly, the first layer image 154b is set to correspond to a part of the first layer 162b of the index block 160a, for example, the 2 × 2 tile image in the upper left, and a pointer to the tile image is set. A null pointer is set for the other tile areas in the index block 160a.

  An area 152c in the header 150 includes images 154f, 154g, and 154h of the third hierarchy of the fifth hierarchy, the sixth hierarchy, and the seventh hierarchy, but these images have more tiles than the number of tile areas that can be defined by one index block 160. Since it consists of images, as shown in FIG. 8, the image plane is divided so that a part of each layer corresponds to the 0th layer, the 1st layer, and the 2nd layer of the index block 160c. A plurality of layers of images belonging to each index block are divided into regions having the same resolution and different resolution.

  Note that the area division mode of the hierarchical structure shown in FIG. 8 is merely an example, and may be determined as appropriate depending on the structure of the original hierarchical data, the data size of a desired index block, and the like. As will be described later, the division mode of the hierarchical data may be changed according to the update of the original image.

  Next, an aspect of generating an image file including hierarchical data having the data structure described so far will be described. This aspect can also be realized by the same apparatus configuration as the information processing system shown in FIG. 1 and the information processing apparatus 10 shown in FIG. FIG. 9 shows a configuration of the control unit 100b having a function of generating an image file to be displayed in the present embodiment. The control unit 100b may have a function for displaying an image as shown in the control unit 100a of FIG. 7, but the illustration is omitted here. On the other hand, you may provide the information processing apparatus 10 provided with the control part 100b which has only the function shown in FIG. 9 separately from the information processing apparatus which exhibits a display function.

  The control unit 100b reads out the image data stored in the hard disk drive 50 and hierarchizes the image layer generation unit 120, the image division unit 122 that divides each layer image into tile images, and analyzes the images in each layer for redundancy. Redundancy detection unit 124 to detect, header / index block generation unit 126 to generate header and index block data in consideration of redundancy, and finally output image file including tile image, header, and index block Includes an image file generation unit 128 for generating.

  The image hierarchy generation unit 120 reads data of an image to be created from the hard disk drive 50. This image data may be data of one image having a certain resolution. The image data to be processed may be specified by the user via the input device 20 as shown in the figure, or a request from another functional block (not shown) that acquired the original image may be accepted. The image hierarchy generation unit 120 further generates image data having a hierarchical structure including the original image by generating image data obtained by stepwise reducing the read image data to a predetermined resolution.

  The image dividing unit 122 generates tile image data by dividing each layer image into a predetermined size. The generated tile image is stored in the main memory 60. At this time, the position of each tile image in the original image is managed by assigning an identification number or the like.

  The redundancy detecting unit 124 detects the image redundancy within the same hierarchy and between the hierarchies by analyzing the image of each hierarchy. As the redundancy within the same hierarchy, for example, a case where data of the same tile image can be used across a plurality of tile areas is conceivable. Further, as redundancy between layers, there may be a case where the difference in appearance between an image obtained by enlarging an image of a low resolution layer and an image of a high resolution layer is not large. Such a redundant region can be displayed by diverting the data of a certain tile image without having the tile image data individually. In this way, image data compression is realized. Specific methods will be described later.

  The header / index block generation unit 126 creates the above-described header and index block data. As described above, in an image in which redundancy is not detected, a pointer to one of index blocks is set for all areas defined in the header, and tile images are defined for all tile areas defined in the index block. A pointer to is set. On the other hand, for image areas having redundancy, tile image data sharing is realized by setting a null pointer in the header or index block.

  The image file generation unit 128 reads the tile image data pointed to by the pointer set in the index block and connects them in order to generate final image data. Then, an image file including image data, a header, and an index block is generated and used as final output data. As described above, the size of the image data can be efficiently compressed by detecting the redundancy of the image and using the data of one tile image for drawing a plurality of regions. In the present embodiment, since the same image with different resolution is included in the image data, even if the images are in different layers, the data of the same tile image can be used to compress the data at a high rate.

  FIG. 10 is a diagram for explaining the relationship between an original image and a tile image when the image has redundancy. In the figure, an image 180a and an image 180b are two layers of images representing the same region. That is, the image 180a is obtained by reducing the image 180b. The grids shown on the image 180a and the image 180b are boundary lines when dividing into tile images. In the figure, for the purpose of explanation, identification numbers are assigned to the tile regions of the images 180a and 180b and the tile image 170, respectively.

  In the figure, it is assumed that an image 180a and an image 180b are drawn with an ellipse and a triangle, and the other areas are filled with a single color as a background. The data of the tile image 170 cut out from each area of such an image is stored in the main memory 60 or the like independently of the original image arrangement. Here, the tile area “4”, which is a single color fill area in the image 180a, corresponds to the tile image “4” cut out from the tile area “4”. In such a case, in the index block 160, a pointer to the tile image “4” is set for the tile area “4”.

  Since the image 180b is a high-resolution image of the image 180a, the correspondence between the tile regions of the two images can be easily derived by the enlargement ratio. In the figure, for example, the tile area “4” of the image 180a corresponds to the tile areas “15”, “16”, “19”, and “20” of the image 180b. Therefore, if the tile area “4” of the image 180a is a single-color filled area, it can be specified that all the tile areas “15”, “16”, “19”, and “20” of the image 180b are filled areas. That is, the tile image “4” is sufficient to display these areas.

  Similarly, when the area that becomes the filled area when the image 180a is enlarged to the resolution of the image 180b is determined by scanning the image 180a, the lower half area of the tile area “3”, that is, the tile area “ It can be seen that “17” and “18” are filled areas. Therefore, these areas can also be displayed by using the tile image “4”. By sharing the tile image in this way, data compression of the tile image becomes possible. As will be described later, the header and index block data can be compressed by preventing the high-resolution image area that can be displayed by the low-resolution tile image from having a specific pointer.

  Since the image data used in the present embodiment targets the same image having a plurality of resolutions, the above-described redundancy detection process is repeated in order from the smallest resolution image to obtain a high resolution large size image. However, it is possible to specify a region having redundancy while suppressing the calculation load. In the above description, an example in which a single-color filled area is detected and tile image data is shared by the same filled area has been described. It is also effective in areas. These areas are detected from a low resolution image by color histogram or frequency analysis for each area.

  FIG. 11 is a diagram illustrating another example of the relationship between the original image and the tile image when there is redundancy. The image 180a and the image 180b are the same as those shown in FIG. In the figure, the tile area “2” of the image 180a corresponds to the tile areas “7”, “8”, “11”, and “12” of the image 180b. Since the image 180b is a higher-resolution image than the image 180a, the four tile images in these regions generally include more information than an image obtained by simply enlarging the tile image “2” cut out from the region “2”. It is.

  However, when the original image is a photograph or the like and the areas “7”, “8”, “11”, and “12” of the image 180b are not in focus, there is a large difference from the enlarged image of the tile image “2”. There may not be. In such a case, when displaying the areas “7”, “8”, “11”, “12”, the tile image “2” is used instead of the tile image cut out from the area “7”, “8”, “11”, “12”. Even if it is enlarged and displayed, an image that is not significantly different from the original image can be displayed. Such redundancy is possible only when the difference between the enlarged image of the low-resolution image and the high-resolution image is less than the threshold value, or when the frequency analysis of the high-resolution image is performed, only in the frequency band below the threshold value. It can be detected when it is included.

  FIG. 12 is a diagram for explaining a technique for defining sharing of tile images based on redundancy as described above using a header. At this time, an area that can be displayed using an image of a hierarchy included in another area in the area of the hierarchical structure defined by the header 150a is set as a null pointer without setting a pointer to a specific index block. In FIG. 12, in the structure of the header 150a, a pointer to a specific index block, that is, a valid pointer is set in the shaded area, and a null pointer is set in the white area.

  When the image area 156 is displayed in the hierarchical data having such a header 150a, the area including the image is a null pointer. In this case, the virtual space of the hierarchical data is traced in the direction in which the image is reduced, and an area including a reduced image of the same image area 156 and set with a valid pointer is searched (arrow F). In the figure, since a valid pointer is set in the area 152d, the data of the index block 160d pointed to by the pointer is acquired (arrow G).

  The index block 160d specifies a tile area including the image area 156 in the second hierarchy having the highest resolution among the layers in which the data is stored, and obtains a pointer associated therewith. Where the image area 156 is located on the image plane of the second hierarchy of the area 152d can be easily calculated based on the scale ratio of each hierarchy in the hierarchy data, the above tile area is specified based on it.

  Then, the data of the tile image 170c pointed to by the acquired pointer is acquired (arrow H), and a display image of the image area 156 is generated using the acquired data. Here, assuming that the image area 156 has the size of the tile image, to draw the image area 156, as shown in the figure, a partial area 172 of the tile image 170c, which is a reduced image, is enlarged. As described above, the position of the region 172 corresponding to the image region 156 can be easily calculated based on the scale ratio of both.

  Thus, by making it possible to define a null pointer in the header 150a, it is not necessary to prepare index blocks for all the areas defined by the header 150a, and the data compression efficiency is improved. Note that the area where the null pointer is set does not necessarily have to be on the high resolution side, and as shown in FIG. 12, the null pointer is set in a state where the area 152e is sandwiched between the low resolution area and the high resolution area. It can also be set. In this case, when displaying an image included in the region 152e, an image of a layer belonging to the lower-resolution region 152f is enlarged and displayed, and when the resolution of the display image increases and enters the region 152s, the region The tile image is acquired with reference to the index block indicated by the pointer set for.

  In this way, when the display image exceeds a certain resolution, a different image can be displayed. For example, when the resolution is increased, display effects such as the appearance of a kana for a kanji on a newspaper or the appearance of another object in the background of a photograph can be achieved. As described above, in the header, it is possible to set a null pointer for an arbitrary area basically, but it is always effective for the vertex of the hierarchical structure, that is, the area 152g including the image with the smallest resolution. Set the correct pointer. Thereby, even if a null pointer is set for any other region, at least the region 152g can be reached and a display image can be generated by following the hierarchical structure.

  FIG. 13 is a diagram for explaining a technique for defining sharing of tile images based on redundancy as described above by using index blocks. Here, among the image areas belonging to the index block 160e, an area that can be displayed using an image of another layer included in the same index block 160e is a null pointer without setting a pointer to a specific tile image. In FIG. 13, it is assumed that a pointer to a specific tile image, that is, a valid pointer is set in the tile area indicated by shading in the index block 160e, and a null pointer is set in the white area.

  When displaying the image area 158 in the figure, first, the area 152h including the image area 158 is specified in the header 150b, and the index block 160e indicated by the pointer set for the area 152h is acquired (arrow I). When a null pointer is set for an area in the image of the second hierarchy 162g corresponding to the image area 158 among areas where the index block 160e holds data, the hierarchy belonging to the same index block 160e is reduced in the reduction direction. Tracing is performed, and a hierarchy in which a valid pointer is set for the reduced image in the same image area 158 is searched.

  In the example of the figure, the image of the first hierarchy 162h is first reached (arrow J), but a null pointer is set for the area corresponding to the image area 158. Therefore, the image of the 0th layer 162i having a lower resolution is reached (arrow K). In the image of the 0th hierarchy 162i, since a valid pointer is set in the area corresponding to the image area 158, the data of the tile image 170d pointed to by the pointer is acquired (arrow L), and is used for the image area 158. Display images can be generated. Assuming that the image area 158 has the size of the tile image, to draw the image area 158, as shown in the figure, a partial area 174 of the tile image 170d which is a reduced image is enlarged. As described above, the position of the region 174 corresponding to the image region 158 can be easily calculated based on the scale ratio of both.

  In this way, by making it possible to define a null pointer in the index block 160e, it is not necessary to define pointers to tile images for all tile areas defined in the index block 160e. The amount of data in the block itself can be reduced. In addition, even if tile images are updated, the relationship between image enlargement and reduction in the index block does not basically change, so it is not necessary to update pointers to all tile images, and update processing is minimized. be able to.

  Similar to the header, in the index block, an effective pointer is set for all tile areas in the image of the 0th layer 162i having the smallest resolution. In this way, even if a null pointer is set in another arbitrary area, the image of at least the 0th hierarchy 162i is reached by following the hierarchy in the reduction direction, and a display image can be generated. However, among the areas defined in the header, the vertex of the hierarchical structure, that is, the area including the hierarchy with the lowest resolution may have a small number of tile images as described above. In such a case, the corresponding index block An area in which a null pointer is set may be included in all the layers.

  Similarly to the header, in the index block, a specific pointer is set in the high resolution second layer 162g and the low resolution zero layer 162i, and only the first layer 162h having an intermediate resolution is null. You can set the pointer. For example, when displaying the lower left area of the intermediate first hierarchy 162h in FIG. 13, the tile image pointed to by the pointer set for the same area of the low resolution 0th hierarchy 162i is enlarged and displayed. When the resolution of the display image increases until the layer 162g is used, a display image is generated using the tile image pointed to by the pointer set for the same region of the second layer 162g. By doing in this way, like a header, the mode that another image will be displayed if it expands also in the same field is realizable.

  As described above, when the sharing of the tile image is defined by the index block, detailed setting can be performed in units of tile images, compared to the case where the sharing of the index block is defined by the header. 12 and 13 show a case where a null pointer is set in either the header or the index block, but even if a null pointer is set in both, the index block and the index block and You can define sharing of tile images. Moreover, in the above-mentioned example, the aspect which shares a tile image between different hierarchies was implement | achieved by setting a null pointer to a header or an index block. On the other hand, when sharing tile images in the same hierarchy, a pointer to the same index block is set for a plurality of areas including the same hierarchy in the area of the hierarchical structure, or multiple tile areas in the index block are set. A pointer to the same tile image may be set.

  Next, the operation of the control unit 100b having a function for generating an image file will be described. FIG. 14 is a flowchart illustrating a processing procedure in which the control unit 100b illustrated in FIG. 9 generates an image file. First, when an input for designating image data is made by the user or another processing module, the image hierarchy generation unit 120 reads the designated image data from the hard disk drive 50 (S50). Then, image data having a predetermined resolution is generated by performing a general reduction process, and hierarchical data composed of image data of a plurality of resolutions is generated (S52). Next, the image dividing unit 122 divides each layer image into tile images and stores them in the main memory 60 (S54). At this time, the position on the image of each tile image and the address indicating the storage area of the main memory 60 are recorded in association with each other, which is used for generating an index block.

  In S50, a file in which tile images are recorded in advance may be read out. In this case, the hierarchical data generation process in S52 and the image division process in S54 can be omitted. If the offset value of the address of each tile image from the head address of the file and the position on the image are included in the file, the head address in the main memory 60 of the file read in S50 is recorded in S54. Thus, the relationship between the address of each tile image and the position on the image can be acquired.

  Next, the redundancy detecting unit 124 confirms the presence or absence of redundancy as described above (S58). Specifically, the image of the Nth layer is scanned to check whether or not there is a region composed of a single color in the region corresponding to the size of the tile image of the N + 1th layer image. This is repeated in the direction of higher resolution. In addition, the redundancy detection unit 124 is an N + 1 layer image obtained by performing a difference image between an image obtained by enlarging an image of the N layer up to the size of the image of the N + 1 layer and an image of the N + 1 layer and frequency analysis of the image of the N + 1 layer. Also, it is confirmed whether or not there is an area containing only the same level of information as the N-layer image. As a confirmation method, a method generally used in the field of image processing may be appropriately applied.

  Next, the header / index block generation unit 126 creates a header and an index block in consideration of redundancy (S60). Specifically, if an image belonging to the low resolution area can be enlarged and displayed for each area defined in the header, a null pointer is set for the area, and the corresponding index block is set for the other area. Generate and set a pointer to each in the header. Further, in each index block, a null pointer is set for an area in which a low resolution image can be enlarged and displayed among the images of a plurality of layers belonging thereto, and tiles cut out from the area for other areas Set a pointer to the image.

  Next, the image file generation unit 128 creates an image file for final output (S62). Here, when the redundancy detection unit 124 does not detect redundancy in S58, all the tile image data generated by dividing in S54 is read from the main memory 60, and the image is displayed together with the header and index block generated in S60. A file. When redundancy is detected, only the data of the tile image pointed to by the pointer set in the index block generated in S60 is read from the main memory 60, and is set as an image file together with the header and index block generated in S60.

  Next, an operation when an image is displayed using the image file generated in this way will be described. FIG. 15 is a flowchart showing a processing procedure for displaying an image using an image file in the present embodiment. This processing procedure can be realized by the control unit 100a shown in FIG. The flowchart in the figure mainly describes the process of reading out the necessary data from the image file in response to a display area change request, decoding and displaying it, but loading from the hard disk, prefetching of the necessary data, The basic flow such as writing is the same as that shown in FIG.

  First, when the user inputs a display area change request via the input device 20 while a part of the image is displayed on the display device 12, the input information acquisition unit 102 receives the request (S70). Then, the tile image specifying unit 110 derives a requested change amount of the display image and determines frame coordinates to be newly displayed based on the derived change amount (S72). The amount of change in the display image is the amount of movement in the vertical and horizontal directions and the amount of movement in the depth direction in the virtual space, and the frame coordinates to be displayed are the frame coordinates of the display area displayed so far and the amount of change derived Can be determined.

  Next, the tile image specifying unit 110 specifies the area to which the frame coordinates belong by referring to the header, and checks whether a valid pointer to the index block is set for the area (S74). If a valid pointer is set (Y in S74), the index block pointed to by the pointer is acquired (S78). If a null pointer is set (N in S74), the hierarchical structure is traced in the image reduction direction to search for an area where a valid pointer is set, and the pointer set for the detected area points to An index block is acquired (S76, S78).

  Further, in the index block, it is confirmed whether a valid pointer to the tile image is set for the area corresponding to the frame coordinates (S80). If a valid pointer is set (Y in S80), information that uniquely defines the tile image such as the address and identification number of the tile image pointed to by the pointer is specified (S84). If a null pointer is set (N in S80), the hierarchy in the index block is traced in the image reduction direction, the hierarchy in which a valid pointer is set is searched, and the hierarchy is set for the corresponding area of the detected hierarchy. The tile image pointed to by the pointer is identified (S82, S84).

  When the decoded data of the tile image is not stored in the buffer memory 70 (N in S86), the decoding unit 112 reads the tile image data from the main memory 60 and decodes it (S88, S90).

  When searching for an area where a valid pointer is set in the header in S76, or when searching for a hierarchy where a valid pointer is set in the index block in S82, the specified tile image is enlarged and drawn. There is a need to. For this reason, when a tile image is specified by following such a process, the tile image specifying unit 110 gives information to that effect to the decoding unit 112. The decoding unit 112 determines whether the tile image needs to be enlarged based on the information (S92).

  When enlargement is necessary (Y in S92), the decoding unit 112 enlarges the tile image based on the frame coordinates acquired from the tile image specifying unit 110, and stores the necessary area in the buffer memory 70 (S94). This process is similarly performed even if the corresponding tile image has been decoded (Y in S86). Regardless of the presence or absence of the enlargement process, the display image processing unit 114 draws a new area to be displayed in the tile image in the frame memory (S96). The drawing process includes a process for enlarging or reducing the data stored in the buffer memory 70 in accordance with the required resolution.

  Next, the data structure of pointers in the header and index block will be described. For example, when hierarchical data composed of 11 hierarchies is grouped into index blocks of 4 × 4, 8 × 8, and 16 × 16 areas as shown in FIG. 8, 4162 areas are defined in the header. become. A maximum of 336 tile areas are defined in one index block. Since a pointer is set for each of these elements, it is desirable to have a data structure that enables efficient retrieval of the pointer. The pointer data structure may be a fixed-length array, an associative array, or a tree structure.

  When the header pointer is a fixed-length array, an identification number is assigned to each area, and the fixed-length pointer array is accessed using this as an index. In this case, since the pointer can be acquired as soon as the area can be specified, editing and search processing can be performed quickly. In the case of an associative array, an identification number is assigned to each area, and the pointer associative array is accessed using the identification number as a key. In this case, since only a valid pointer needs to be held, the data size of the header can be suppressed.

  In the case of a tree structure, pointers that connect the regions are further defined, and a desired region is searched by following the pointers between the regions from the vertices of the hierarchical structure of the image. In this case, as described later, when a hierarchy is added to the vertex, it is only necessary to update the pointer near the vertex. Since the effectiveness varies depending on the data structure of the pointer as described above, the selection is appropriately made according to points to be emphasized such as memory cost and processing cost. Further, a method generally used in data search such as a B-tree structure may be introduced as appropriate. Since the pointer of the index block has the same mode, an optimal method may be selected in consideration of the processing speed.

  The pointer to the index block and the pointer to the tile image may include the name of the file in which the target index block and tile image are recorded, location information of a server connected to the network, and the like. Thereby, it is possible to include a plurality of files and images from a site in one hierarchical data, or to share one file with a plurality of hierarchical data.

  Next, a description will be given of a case where an image file composed of a header, an index block, and a tile image as described above is modified or altered. This aspect can also be realized by the same apparatus configuration as the information processing system shown in FIG. 1 and the information processing apparatus 10 shown in FIG. FIG. 16 shows a configuration of the control unit 100c having a function of correcting an image in the present embodiment. The control unit 100c may have the function for displaying an image shown in the control unit 100a in FIG. 7 and the function for generating an image file shown in the control unit 100b in FIG. The illustration is omitted. On the other hand, the information processing apparatus 10 including the control unit 100c having only the function illustrated in FIG. 16 may be provided separately from the image processing apparatus that exhibits the display function and the image file generation function.

  The control unit 100c includes an image file acquisition unit 318 that acquires an image file to be corrected, an update information acquisition unit 320 that acquires update information including an area updated by correction and image data of an update portion, and an updated tile image. A tile image generation unit 322 for generating a header, an index block update unit 324 for updating a header and an index block, a display image control unit 326 for displaying an image being corrected, and an image file generation unit for generating a corrected image file 328.

  The image file acquisition unit 318 receives designation input of the image to be corrected by the user, reads the corresponding image file from the hard disk drive 50, and stores it in the main memory 60. As described above, the image file includes header, index block, and tile image data. The update information acquisition unit 320 acquires update information input via the input device 20 while viewing the correction target image displayed on the display device 12 by the user. As described above, the update information includes an area to be updated and image data after correction in the area. Specific examples will be described later.

  The tile image generation unit 322 identifies a tile image that needs to be updated when the updated image is applied to the region to be updated, and generates a new tile image. Here, when the area to be updated only covers a part of the tile image, the original tile image is read from the main memory 60, and a new tile image is generated by overwriting only the area to be updated. When all the tile images are included in the region to be updated, a new tile image is generated by cutting out the corrected image. At this time, a new tile image is generated for the areas to be updated in all hierarchies constituting the hierarchy data. However, only a specific hierarchy may be updated depending on the designation of the user. In this case, it is possible to realize a mode in which different images are displayed within a specific resolution range.

  The generated tile image is stored in the main memory 60. At this time, the original image data stored in the main memory 60 by the image file acquisition unit 318 is left as it is, and the newly generated tile image is stored in another storage area. Next, the header / index block update unit 324 rewrites the pointer set for the area to be updated in the index block stored in the main memory 60 to point to the newly generated tile image. If there is no index block corresponding to the area, a new index block is generated, and the null pointer setting in the header is rewritten with a pointer indicating the generated index block.

  The display image control unit 326 may be configured by functional blocks included in the control unit 100a illustrated in FIG. 7, but is not illustrated here. The display image control unit 326 displays the image to be corrected by the same processing procedure as described in FIG. 15, but the header and the index block are updated as described above according to the correction of the image by the user. Accordingly, the tile image used when drawing the image changes to a new one. In this way, it is possible to correct or change the image while confirming the display image.

  In this embodiment, the newly generated tile image is added to another storage area without updating the original tile image data, and the image is updated by changing the reference destination by the index block or the header. In this way, it is possible to display an image in the middle of correction with high responsiveness and lower processing cost compared to the case of searching and overwriting the tile image data corresponding to the correction area in the original tile image. Also, it is possible to easily return to the original image during the correction.

  The image file generation unit 328 overwrites the original tile image data with the newly generated tile image data, for example, when the user finishes the correction. Alternatively, the newly generated tile image data is stored as a separate file from the original tile image data. Then, an updated index block and header are added to form an image file.

  Next, a specific example of processing for each data when an image is corrected or altered by the above-described mechanism will be described. FIG. 17 shows an example of correcting an image as a premise for explanation. In the figure, an ellipse and a triangle are drawn in the image 190 before correction. Consider a case where this is corrected and a star figure 194 is added to the upper right of the image as shown in the corrected image 192. FIG. 18 schematically shows changes in the pointers of the index block in such correction. In the figure, the pointer is indicated by an arrow, but only representative arrows are shown in order to avoid complication of the figure.

  First, before modification, each area defined by the header 150 is set with a pointer to one of the index blocks 160 or a null pointer. A pointer to one of the tile images 170e generated before the correction is set for each tile area of a plurality of layers belonging to the index block 160 pointed to by the pointer, as indicated by a dashed arrow. If the upper right region of the image 190 before correction in FIG. 17 is a monochrome background, the region shares one tile image. In FIG. 18, many broken arrows indicate one tile image 170f. Of course, pointers to individual tile images are set in the other areas.

  When a star graphic 194 is added to such a background portion as in the corrected image 192 in FIG. 17, a new tile image in which a star graphic of each resolution is added as a tile image in the corresponding region of each layer. 170g is generated and stored in a storage area different from the original tile image 170e. Then, in the index block 160, all pointers to the tile areas related to the area to be updated are updated to point to any tile image of the new tile image 170g. In FIG. 18, the updated pointer is indicated by a one-dot chain line arrow. Since the area to be updated is no longer the background after the update, as shown in the figure, pointers to the respective tile images are set for the respective tile areas. Further, the area other than the area to be corrected may remain pointing to the original tile image 170e.

  Next, when the image added by updating like the star graphic 194 in FIG. 17 has a higher resolution than the highest resolution image of the hierarchical data so far, a procedure for adding a hierarchy to the high resolution side is performed. explain. FIG. 19 is a diagram for explaining processing when a hierarchy is added to the high resolution side. In the figure, it is assumed that the image before correction is hierarchical data 200 including a 0th hierarchy 154i, a first hierarchy 154j, a second hierarchy 154k, and a third hierarchy 154l. Here, in the header, it is assumed that valid pointers to the index block are set for all areas, and in FIG.

  When such an image is corrected by adding a high-resolution image to the upper right area as shown in FIG. 17, a tile image of the update target area is newly generated for each layer as described above. Here, since the added image has a higher resolution than the third hierarchy 154l having the highest resolution in the original hierarchy data, it is necessary to add a new hierarchy. In the figure, the layer to be added is the fourth layer 154m. A region where a new tile image is to be generated is shown in black in the figure.

  In this example, since the correction of the image with the high resolution image is local, when the region other than the region to be updated is displayed at the resolution of the fourth layer 154m, it has the highest resolution among the original layer data. The image of the third hierarchy 154l is enlarged and displayed. Therefore, a null pointer is set for a region other than the update target region in the fourth layer 154m, and the tile image of the third layer 154l is used. The area where the null pointer is set is shown in white in the figure.

  As described above, there are a method of setting in the header and a method of setting in the index block in sharing the tile image. When the number of hierarchies combined in one index block is fixed, a hierarchy corresponding to the fourth hierarchy 154m to be added may already be defined in the existing index block. Since the fourth layer 154m does not exist in the original layer data, a null pointer is set for the same layer of the index block.

  In such a case, in the existing index block, the null pointer set for the update target area of the fourth hierarchy 154m is updated with a pointer to the newly generated tile image. The area other than the area to be updated remains a null pointer. Similarly, the index block pointer is updated for each layer of low resolution.

  FIG. 20 is a diagram for explaining changes in the header and the index block when there is no index block to which a layer corresponding to the fourth layer 154m to be added belongs. In the header 150 in the figure, an area 152i indicates an area defined for the hierarchical data before correction. Of course, there may be a plurality of areas defined by the header 150, but in the figure, they are collectively shown as one area 152i. The 0th hierarchy 154i, the 1st hierarchy 154j, the 2nd hierarchy 154k, and the 3rd hierarchy 154l which existed before correction belong to the area | region 152i. A pointer to the index block 160f is set for each area. A pointer to the existing tile image 170h is set for each tile area defined by the index block 160f (dashed arrow).

  Here, when adding the fourth layer 154m, first, in the header 150, a new area including the layer is added. However, as shown in FIG. 20, a valid pointer is set only for the region 152j including the region to be updated, and a null pointer is set for the other regions. Further, a new index block 160g is generated, and a pointer to the index block 160g is set for the area 152j. If the hierarchy to be added is only the fourth hierarchy 154m, out of the three hierarchies belonging to the newly generated index block 160g, the pointer to the tile image may be set at the lowest hierarchy, and the other hierarchies are Set a null pointer for all areas.

  In the newly generated index block 160g, a pointer is set so as to point to one of the newly generated tile images 170i with respect to the tile area included in the area to be updated (dashed line arrow). Similarly, the existing index block 160f is updated so that the pointer set for the tile area included in the area to be updated points to one of the newly generated tile images 170i. When the image file is finally generated, the newly generated index block 160g may be added to the end of the original image file or may be another file. Thereby, the images before and after the update can be displayed.

  Next, a procedure for extending the size of an image by adding a new area such as a background or a drawing area to an existing image will be described. FIG. 21 is a diagram for explaining processing for adding a new area to an existing image. In the figure, it is assumed that the image before addition is hierarchical data 204 including a 0th hierarchy 154i, a first hierarchy 154j, a second hierarchy 154k, and a third hierarchy 154l. In the area to which each hierarchy belongs, a valid pointer to the index block is set in the header.

  When a new area is added to such an image, hierarchical data 206 is generated by adding a new area to the image of each hierarchy at a size corresponding to each resolution. In the example of FIG. 21, the hierarchy data 206 includes a 0th hierarchy 154n, a first hierarchy 154o, a second hierarchy 154p, a third hierarchy 154q, and a fourth hierarchy 154r. The fourth layer 154r and the third layer 154q can be generated by adding the tile image of the additional area shown by shading to the tile image of the original image shown by white. Depending on the size of the second layer 154p, a tile image indicated by shading in which the additional area and the original image are mixed is added to the area indicated by white where the original tile image can be used as it is. The first hierarchy 154o and the 0th hierarchy 154n are all shaded because the original image and the additional area are mixed in all the tile images constituting the first hierarchy 154o and the 0th hierarchy 154n.

  Thus, when a new area is added to an image, the original tile image can be used as it is depending on the hierarchy. Using this property, the original index block is used as it is, and a new index block is generated only for a necessary area such as an additional area. FIG. 22 is a diagram for explaining changes in the header and the index block when a new area is added. In the header 150 of the figure, a white triangle indicates a region 208 where the original tile image can be used, and a shaded portion indicates a region 210 that is changed by addition or addition.

  The broken lines indicate the 0th hierarchy 154n, the 1st hierarchy 154o, the 2nd hierarchy 154p, the 3rd hierarchy 154q, and the 4th hierarchy 154r in FIG. 21, and the area where the original tile image can be used in each hierarchy, and additional or The area to be updated is determined by the addition, resulting in a structure as shown in FIG. Since the area defined by the header is generated sequentially from the top of the hierarchical structure, when the header 150 changes in this way, the area delimiter changes.

  However, by devising the area demarcation as will be described later, even if there is a change that increases the hierarchy at the top of the hierarchy data as shown in FIG. 22, the area defined before the change is saved as much as possible. As a result, the existing index block can be used as it is. In FIG. 22, before adding the area, a pointer to the index block 160h is set for each area (not shown) of the area 208. In each tile area defined by the index block 160h, a pointer to an existing tile image 170h is set (broken arrow).

  After the area is added, a new index block 160 i corresponding to the area to be added is generated, and a pointer to the index block 160 i is set for the new area 210 in the header 150. Here, the area 210 may have a plurality of areas defined in the header 150, but is not shown. In the generated index block 160i, a pointer to one of the new tile images 170k required by adding the area is set for each tile area (dashed line arrow). Here, the new tile image 170k is a tile image of the region to be added itself, or a tile image in which the region to be added and the original image are mixed, which are indicated by shading in FIG. At this time, if the area to be added is a monochrome background, one tile image can be shared.

  On the other hand, if the area defined originally in the header is not changed in how the area is divided, the set pointer and the index block pointed to by it can be used as they are. Therefore, there is no need to update the pointer to the tile image set in the index block. In the above example, the right and lower regions of the image are added, but the region is divided so that the existing index block can be used even after the image is expanded in any direction. As a result, changes in the header and index block can be minimized.

  FIG. 23 is a diagram illustrating a method of dividing an area defined in a header when adding a hierarchy to the vertex side of a hierarchical structure such as when expanding the image size as described in FIGS. 21 and 22. . First, in the first hierarchical data 230, an area obtained by dividing the entire 0th to 1st hierarchies by the first area 152k, the entire 2nd to 4th hierarchies by the second area 152l, and the 5th to 7th hierarchies by areas on the image in order. The area is divided into a third area 152m, a fourth area 152n,. The configuration of this area is the same as that shown in FIG.

  As described above, the number of tile areas defined by one index block is prepared equally regardless of the index block, but the first area 152k is a 0th layer 2 × 2 tile image composed of one tile image. Since the index block 160j corresponding to the area is composed of only the first layer configured, only that number of tile areas is used, and null pointers are set to the other tile areas and invalidated.

  In the index block shown in FIG. 23, the number of effective tile areas in the horizontal direction is indicated by a rectangle for each layer. Accordingly, the index block 160j corresponding to the first region 152k is shown as one and two rectangles. Since the index blocks 160k, 160l, etc. corresponding to other areas such as the second area 152l and the third area 152m can make all 4 × 4, 8 × 8, and 16 × 16 tile areas effective, there are four. 8 and 16 rectangles are shown.

  Here, with respect to the added hierarchy data 232 on the top side of the hierarchy data 230, the added hierarchy is further included in the first area 152o without changing the hierarchy that originally belonged to the first area. Since the index block 160j corresponding to the original first area 152k is used for only two hierarchies out of the three hierarchies prepared as the data area, the hierarchies are added only by using the hierarchies that have been invalidated only in the first area 152o. Can be absorbed. Therefore, the second to fourth hierarchies (third to fifth hierarchies after the addition of the hierarchy) included in the second area 152l in the original hierarchy data 230 can belong to the same area as they are. However, since the image of each layer is expanded, a new area is defined for the expanded portion and an index block is generated. The same applies to the lower layers.

  Here, for the added hierarchy data 234, one more hierarchy is added to the apex side of the hierarchy data 232, and the added hierarchy is further included in the first area 152p without changing the hierarchy that originally belonged to the first area. In this case, only the first area 152p has four layers, but the tile area prepared in the corresponding index block 160m is larger than the number of tile images in the first area 152p. This information is satisfied by the index block 160m. Then, the hierarchy below that can belong to the same area as that belonging to the first hierarchy data 230.

  In the hierarchy data 236 that is further added to the hierarchy data 234, the added hierarchy is divided into four areas that belong to the first area 152p in the hierarchy data 234, and the first area is divided into two areas. 152q and the second region 152r. At this time, by setting the first area 152q to the 0th to 1st hierarchies and the second area 152r to the 2nd to 4th hierarchies, the index block 160n corresponding to the first area 152q becomes the first area 152k of the first hierarchy data 230. Like the index block 160j corresponding to, it consists of only one tile image layer and 2 × 2 tile image layers. The second area 152r is the same as the second area 152l of the hierarchical data 230.

  The lower hierarchy can belong to the same area as that belonging to the first hierarchy data 230. By repeating the above changes, even if a hierarchy is added to the vertex of the hierarchy data, the area can be defined as much as possible without changing to other areas, and the existing index block can be used as it is.

  When the image area is expanded as shown in FIG. 21, in the tile image management based on the order of the raster direction, which is generally used in image processing, the tile image of the added area is between the existing tile images. Therefore, it is necessary to rearrange the tile images and reassign the identification numbers. As the number of tile images increases, the load on this process increases. In the present embodiment, as described above, the image area can be expanded only by local data change, so the processing cost is low, and the data display being updated and the final image file generation are realized with high responsiveness. it can.

  Next, operations that can be realized by the above configuration will be described. FIG. 24 is a flowchart showing a processing procedure when the user corrects or modifies an image in the image processing apparatus. First, when the user designates an image file to be corrected, the image file acquisition unit 318 reads the corresponding image file from the hard disk drive 50 and stores it in the main memory 60 (S110). Then, the display image control unit 326 displays the image of the image file on the display device 12 (S112).

  In this state, when the user inputs update information while viewing the image displayed on the display device 12 (Y in S114), the update information acquisition unit 320 accepts the update information, and the tile image generation unit 322 receives a new tile image of the update part. Is stored in the main memory 60 (S116). The input of update information performed by the user is performed, for example, by displaying an image to be newly pasted in the vicinity of the image displayed on the display device 12 and allowing the user to drag it to a desired area with a pointing device. Alternatively, a character or picture may be directly drawn with a pointing device, or a command for adding an area may be input.

  Next, the header / index block update unit 324 appropriately updates the header and index block according to the update content of the image as described above (S118). Then, the display image control unit 326 updates the display of the display device 12 by reading and decoding the tile image data that is newly referred to (S120). The processes from S114 to S120 are repeated until the correction is completed and update information cannot be obtained.

  When it is detected that the correction has been completed by an input from the user or the like (N in S114), the image file generation unit 328 uses the newly created tile image stored in the main memory 60 as the original image file. The image data is reconstructed by being incorporated in the image data (S122). Alternatively, only the newly created tile image is stored in a separate file. Then, it is output as an image file together with the final header and index block data updated in S118 (S124).

  During correction, the header, index block, and tile image are stored in a storage area different from existing data. When the user performs an operation (Undo) to cancel the update of the immediately preceding image, the image can be easily returned to the previous state by returning the reference destination of the data. Similarly, the operation re-execution (Redo) is performed only by changing the reference destination.

  According to the present embodiment described above, hierarchical data is composed of three data sets of a header, an index block, and a tile image. As a result, the tile image can be made independent from the position on the original image, management becomes easier when the image is modified, etc., and the tile image is shared by multiple areas in consideration of image redundancy. Therefore, the data size of the tile image can be efficiently reduced.

  When a tile image is shared by multiple areas, the pointer set in the header or index block is invalidated, and in the invalid area, the hierarchy is traced to the lower resolution side and the valid pointer is set Explore. Thereby, the size of the data itself used for the index can be reduced as compared with the case where the pointer to any tile image is set for all the regions.

  Further, the hierarchical structure is divided into regions by the header, and an index block is associated with each region. At this time, a plurality of hierarchies are included in one area. As a result, data that is likely to be displayed or modified at a time can be handled collectively, and it is easy to avoid an inefficient situation in terms of storage area and processing, such as loading unnecessary data into the main memory. In addition, by fixing the number of tile areas that can be defined in the index block, fragmentation during data caching does not occur, and storage area management is easy. Even if the image is enlarged and the data size is increased, the data to be displayed and corrected can be efficiently accessed by dividing the area of the hierarchical data.

  For example, in the case of hierarchical data composed of 80000 tile images, if the number of tile areas defined by the index block is 400, 200 index blocks are generated. Here, when it is desired to access one tile image, if it is just a list of 80000 tile images, it is necessary to search for them in order, but if the area is divided as in the present embodiment, 1 Searching for one area is equivalent to searching for 400 tile images, and efficient access is possible.

  Similarly, when displaying on a general image display device, the average number of index blocks required for an image to be displayed at one time is one to two, and four or more index blocks are required. It is rare. Therefore, compared with the case where the index to the tile image is held without dividing the area, the index data size required at one time can be reduced, and the main memory can be saved. As a result, there is a high possibility that an aspect in which a plurality of hierarchical data is processed simultaneously can be realized without squeezing the main memory.

  Furthermore, the pointer set in the header and the index block can also set a file in which the index block and the tile image are recorded. By doing so, data read from a plurality of index block files and tile image files can be used even with one hierarchical data, and hierarchical data can be generated and updated more flexibly. Also, if the files are different, it is not necessary to store them in the same storage medium, so that huge image data exceeding 4 GB, for example, can be generated. This makes it possible to process uncompressed images, HDR (High Dynamic Range) images, attribute information, and the like.

  At the same time, a file of data to be used can be shared by a plurality of hierarchical data. For example, images that can be reused as templates, such as magazine headers and footers, can be shared without being retained individually, thereby reducing the data size of each hierarchical data. Not only tile images but also index block data may be shared.

  By applying this, content upgrade using hierarchical data can be easily realized. For example, a file including header, index block, and tile image data is distributed as normal version content. A separate tile image file with a higher resolution is prepared for addition. In the normal version header and index block, a pointer to the tile image for addition is also described. However, unless the tile image is obtained, the pointer is invalid and is not displayed. When the user upgrades the content, an additional file is downloaded, and the pointer to the tile image recorded in the file is valid, so that the additional image is displayed. In this way, the upgrade can be easily performed without changing the normal content file. You may be charged for the upgrade.

Embodiment 2
In the first embodiment, the image file is generated and corrected mainly based on information input by the user. In the present embodiment, an aspect of rendering into hierarchical data by a graphic processor will be described. By introducing hierarchical data into rendering, as described in the first embodiment, it is only necessary to perform processing corresponding to the required resolution, and even a high-definition huge image is efficient. Processing is possible. The present embodiment can be realized by the information processing apparatus 10 shown in FIG. 4 of the first embodiment. FIG. 25 shows in more detail the configuration of the control unit 100 and the main memory 60 in the information processing apparatus 10 according to the present embodiment.

  The control unit 100 includes a main processor 400 and a graphic processor 402. The main processor mainly executes an application program started in the information processing apparatus 10 and controls other functional blocks included in the information processing apparatus 10. The graphic processor 402 executes image processing in accordance with an image processing request from the main processor 400. Data necessary for image processing is stored in the main memory 60. The graphic processor 402 reads out the data and appropriately performs image processing, and then writes the result in the main memory 60 to perform image processing.

  The graphic processor 402 performs rendering using the model data 418 stored in the main memory 60 in accordance with an image processing request from the main processor 400, and updates the color buffer 414 and the Z buffer 416 that are also stored in the main memory 60. In this embodiment, as shown in FIG. 25, each of the color buffer 414 and the Z buffer 416 has a hierarchical data structure. That is, the color value pixel plane that holds color information for each pixel of the drawing target image and the Z value pixel plane that holds depth information from the viewpoint for each pixel are associated with the plurality of resolutions of the drawing target image. It has a hierarchical structure. As described in the first embodiment, the hierarchical data may be composed of headers, index blocks, and tile data. Here, the tile data is data for each tile area obtained by dividing the pixel plane of each layer into a predetermined size.

  The graphic processor 402 performs rendering in accordance with a request from the main processor 400, and updates the hierarchical data to reflect the rendering result of the rendering unit 404 in each layer of the color buffer 414 and the Z buffer 416 stored in the main memory 60. Part 406. The graphic processor 402 further includes a GPU color buffer 408, a GPU-Z buffer 410, and a dirty mask 412 as storage areas temporarily used during rendering. The dirty mask 412 has an area of the same size as the GPU color buffer 408 and the GPU-Z buffer 410, and each pixel holds 1-bit information. When the GPU color buffer 408 and the GPU-Z buffer 410 are updated by rendering, the pixel area is detected by changing the value of the pixel to be updated.

  Next, the operation during rendering of the image processing apparatus having the above configuration will be described. FIG. 26 is a flowchart showing a processing procedure of rendering to hierarchical data. First, the main processor 400 makes a rendering request to the graphic processor 402 (S200). Data necessary for rendering is stored in the main memory 60 as described above. In addition to the model data 418, the color buffer 414 and the Z buffer 416 having a hierarchical structure as described above are also included.

  Then, the rendering unit 404 of the graphic processor 402 identifies the layer and area to be rendered in the color buffer 414 and the Z buffer 416 based on the model data 418 and the like (S202). Next, the rendering unit 404 reads the data of the area from the color buffer 414 and the Z buffer 416 and spreads the data in the GPU buffer including the GPU color buffer 408 and the GPU-Z buffer 410 (S204). Similar to the tile image specifying method described in the first embodiment, the method of specifying tile data from the hierarchy and area to be rendered is searched in the order of header and index block. Therefore, reading is performed in units of tile data.

  At this time, the rendering unit 404 initializes the dirty mask 412 (S206). For example, each pixel value is set to “0”. Then, rendering is performed by a general method using the model data 418, the GPU color buffer 408 and the GPU-Z buffer 410 are appropriately updated, and the pixel value to be updated in the dirty mask 412 is updated to “1” ( S208).

  Then, based on the rendering results in the GPU color buffer 408 and the GPU-Z buffer 410, the hierarchical data of the color buffer 414 and the Z buffer 416 in the main memory 60 is updated (S210). Specifically, first, among the hierarchical data, the tile area including the pixel to be updated is specified based on the pixel value of the dirty mask 412 in the hierarchy and area to be rendered, and the data of the tile area is stored in the GPU color buffer. 408, updated based on the rendering result of the GPU-Z buffer 410.

  At this time, as described in the first embodiment, new tile data is generated, and the pointer set in the index block is updated to a pointer to the new tile data. If frequent updating is expected, the generation of tile data may be omitted and only the pointer may be updated, and the tile data may be generated after the image is confirmed. Next, for the layers other than the layer to be rendered, tile data of a region corresponding to the updated region is newly generated, and the pointer set in the index block is updated with a pointer to the new tile data. . At this time, since the resolution varies depending on the hierarchy, the rendering result is enlarged or reduced and reflected in new tile data.

  When tile data of the rendering target area originally does not exist in the color buffer 414 and the Z buffer 416, such as when rendering an image with a resolution higher than that of the existing hierarchical data, the corresponding area of the upper hierarchical level is enlarged and the GPU is expanded. Cover the color buffer 408 and the GPU-Z buffer 410. Then, after rendering, only the updated tile data is reflected in the color buffer 414 and the Z buffer 416, and the header or index block pointer is updated. Thereafter, tile data is generated and pointers are updated for layers other than the layer to be rendered, as described above. This process is the same as that described with reference to FIG. 19 in the first embodiment.

  Note that when the rendering result is reflected in the hierarchical data in S210, if the area in which the rendering result is expanded to reflect in the hierarchy below the rendering target hierarchy occupies one of the tile areas in the hierarchy, the tile data. Deletes and invalidates the pointer to it. Thereby, the size of tile data can be reduced. At the time of display, as described in the first embodiment, a hierarchy in which the pointer is valid may be searched and an image in the upper hierarchy may be enlarged.

  The color buffer 414 and the Z buffer 416 may hold the header and the index block individually or may share them. In the case of sharing, the index block sets a plurality of pointers for one tile area, that is, pointers to color buffer tile data and Z buffer tile data. Thereby, the data of the header and the index block can be compressed rather than being held individually. When an image is determined after rendering, a pointer to tile data in the Z buffer that is no longer necessary in the index block may be deleted. A target pointed to by a plurality of pointers set in one tile area in the index block is not limited to a color buffer or a Z buffer depending on the contents of the image.

  A Z buffer may be shared among a plurality of color buffers. For example, two color buffer data and one Z buffer data may be prepared, rendering may be performed with one color buffer and Z buffer, and rendering may be performed with the other color buffer and Z buffer. In this case, it is more effective in terms of control to hold the header and the index block individually.

  This embodiment of rendering a color buffer and a Z buffer as hierarchical data can be applied to anti-aliasing. That is, hierarchical data composed of a high resolution hierarchy and a display image hierarchy having a lower resolution is generated. In general supersampling, it is necessary to prepare a storage area for a high-resolution image even for an area where no object exists, such as a monochrome background. On the other hand, by making these buffers hierarchical data as described above, an area obtained by enlarging the low-resolution hierarchy can be excluded from the rendering target of the high-resolution image.

  For example, when the user designates a necessary resolution for each area, only the minimum necessary area can be rendered in each layer, and the use efficiency of the memory is improved. In order to perform frequent tile data update and image display with high responsiveness, the generated tile data need not be compressed. If anti-aliasing is required for the entire image, super-sampling may be performed, and the anti-aliasing method may be switched depending on the ratio of the target area.

  When anti-aliasing is performed using hierarchical data, only the lowest layer image is always rendered, and the reduced image is generated after the image is confirmed, and the rendering is performed at the level specified by the user, and each level corresponds to each level. It is conceivable that the area is also updated. In the former, the frequency of reduction processing is kept low, and in the latter, it is easy to predict the cost of pixel generation processing such as rasterization, shader processing, ROP (Rendering Output Pipeline), and pixel transfer. However, in the latter case, data such as an original image, an enlarged image, and a reduced image are frequently transferred between the graphic processor 402 and the main memory 60, so that a load on the transfer bandwidth is easily applied.

  When rendering hierarchical data as well as in the case of anti-aliasing, a scaling process for enlarging or reducing an image is frequently required in order to reflect the rendering result in each hierarchy. When image data is sent and received between the graphic processor and main memory each time scaling processing is performed, there is a concern that the transfer bandwidth between the two may be compressed. Therefore, in the present embodiment, as shown in FIG. 27, a scaler 422 that performs scaling processing may be built in the main memory 60.

  As a result, the graphic processor 520 and the main memory 60 need only exchange images with the resolution to be rendered, and the scaler 422 reflects the rendering result in each layer in the layer data in the main memory 60. To do. As a result, compression of the transfer bandwidth between the graphic processor 520 and the main memory 60 can be prevented, and hierarchical data can be rendered efficiently. The scaler 422 is not built in the main memory 60, and the same effect can be obtained by connecting to the main memory 60 by a bus different from the bus between the graphic processor 520 and the main memory 60.

  The above-described embodiment can be realized using a general graphic processor. Here, in a general graphic processor, when addressing an internal frame buffer, that is, the above-described GPU buffer, in order to suppress the number of reading and writing from the main memory, a memory address close to a grouped area on the screen is assigned. Take the technique. However, there is a restriction that the area used for the GPU buffer must be a continuous address space. That is, it is not possible to cover areas with discontinuous addresses. 28 and 29 show the state of processing at that time.

  First, in FIG. 28, it is assumed that addresses “11” to “18” are assigned when a partial region 430 of the third layer image of the layer data is to be rendered. In this case, the graphic processor 402 can lay out the area 430 in the GPU buffer. Then, as shown in FIG. 29, it is assumed that a part of the region 432 in the same third hierarchy is to be rendered. The area 432 includes areas to which the addresses “11” and “12” have been assigned in FIG. In this case, addresses “19” to “24” are allocated to other areas of the area 432. When address assignment is performed in this way, the area 432 includes areas where the addresses are discontinuous, and the graphic processor 402 cannot spread the area in the GPU buffer.

  For this reason, it is necessary to copy the area 432 once to another storage area and assign new continuous addresses (addresses “50” to “57” in FIG. 29) to spread the GPU buffer. In the present embodiment, in order to reduce such processing costs, a memory controller (not shown) of the main memory 60 may be provided with a table that associates the position of the area with the address to be allocated. In the case of the area 432 in FIG. 29, the position of the area is expressed by “0” to “7” in raster order, and “0” = “19”, “1” = “20”, “2” = “21”, “ 3 ”=“ 22 ”,“ 4 ”=“ 23 ”,“ 5 ”=“ 24 ”,“ 6 ”=“ 11 ”,“ 7 ”=“ 12 ”.

  In this way, even when rendering an area to which discontinuous addresses are allocated, address conversion is performed in the main memory 60, so that a copy for spreading as shown in FIG. 29 is not necessary. Further, as shown in FIG. 27, if the main memory 60 has a built-in scaler 422, many processes for generating and updating hierarchical data, including generation of enlarged and reduced images, can be closed in the main memory 60. it can.

  By enabling rendering of hierarchical data as in the present embodiment, random access to tile data by header and index block, multiple hierarchical data by dividing the header to reduce the memory cost for each hierarchical data Thus, it is possible to obtain features such as simultaneous processing, addition, deletion and updating of tile data by rendering. These features are equivalent to the texture requirements of random access, multi-texturing, render target. Therefore, hierarchical data can be handled in the same way as texture, and the hierarchical data can be applied to a wide range of uses using computer graphics.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

DESCRIPTION OF SYMBOLS 1 Information processing system, 10 Information processing apparatus, 12 Display apparatus, 20 Input apparatus, 38 Tile image, 50 Hard disk drive, 60 Main memory, 100 Control part, 102 Input information acquisition part, 106 Load block determination part, 108 Load part, 110 tile image specifying unit, 112 decoding unit, 114 display image processing unit, 120 image layer generating unit, 122 image dividing unit, 124 redundancy detecting unit, 126 header / index block generating unit, 150 header, 160 index block, 170 tile Image, 318 image file acquisition unit, 320 update information acquisition unit, 322 tile image generation unit, 324 header / index block update unit, 326 display image control unit, 328 image file generation unit, 40 Main processor, 402 graphics processor, 404 rendering unit, 406 hierarchical data update unit, 408 GPU color buffer, 410 GPU-Z buffer, 412 dirty mask, 414 color buffer, 416 Z buffer, 418 model data, 422 scaler, 520 graphic processor .

Claims (13)

An image processing request unit that issues an image processing request including information for drawing an image to be newly displayed by executing a program;
A color value pixel plane that holds color information for each pixel of an image to be drawn and a Z value pixel plane that holds depth information from the viewpoint for each pixel are layered according to the multiple resolutions of the drawing target image A hierarchical data storage unit for storing the hierarchical color buffer and the hierarchical Z buffer,
An image processing unit that receives the image processing request, executes image processing, and updates the hierarchical color buffer and the hierarchical Z buffer;
The image processing unit
A drawing color buffer and a drawing Z buffer for recording data of a drawing target hierarchy and area read from the hierarchy color buffer and the hierarchy Z buffer, respectively;
Based on the image processing request, the hierarchy corresponding to the resolution and area of the image to be newly displayed and the area of the hierarchy color buffer and the hierarchy Z buffer are specified as the drawing target hierarchy and area , respectively. reads the data to the drawing color buffer and the drawing Z buffer, performs image processing operations on the basis of the image processing request, a drawing unit that updates the read data,
The data of the original hierarchy and area in the hierarchical color buffer and the hierarchical Z buffer are updated with the updated data in the drawing color buffer and the drawing Z buffer, and the updated data is subjected to scaling processing. A hierarchical data update unit that updates the data in the corresponding area of the other hierarchy ,
An image processing apparatus comprising: The hierarchical color buffer and the hierarchical Z buffer include tile data obtained by dividing a pixel plane of each hierarchy into a predetermined size, a tile area that is an individual area after division of each hierarchy, and the tile area as a drawing target. Index data that associates the tile data to be used when
The layer data update unit newly generates tile data for each layer including a drawing target region for each layer, and newly adds tile data associated with the tile region including the drawing target region in the index data. The image processing apparatus according to claim 1, wherein the data of each layer is updated by using the tile data. The image processing unit further includes a pixel update mask that holds, for each pixel, information on whether or not the pixel values of the drawing color buffer and the drawing Z buffer have been updated in the image processing calculation by the drawing unit. ,
The hierarchical data update unit refers to the pixel update mask, specifies a tile area including an updated pixel in the tile area, and generates tile data of the tile area. An image processing apparatus according to 1. When there is no tile data corresponding to the drawing target area in the drawing target hierarchy in the hierarchical color buffer and the hierarchical Z buffer, the drawing unit enlarges a corresponding area of another hierarchy having a resolution lower than that of the hierarchy. Read and update the drawing color buffer and the drawing Z buffer,
The hierarchical data update unit adds data that associates a tile area of a drawing target hierarchy including the drawing target area with tile data newly generated for the tile area to the index data. The image processing apparatus according to claim 2.   The hierarchical data update unit deletes tile data associated with the tile area in the index data when the entire area to be rendered is included in the area to be rendered in a hierarchy having a higher resolution than the hierarchy to be rendered. The image processing apparatus according to claim 4.   The index data is obtained by dividing a hierarchical structure including the hierarchical color buffer and the hierarchical Z buffer in a virtual space defined by a pixel plane of the hierarchical color buffer and the hierarchical Z buffer and a resolution axis perpendicular thereto. 6. The image processing apparatus according to claim 2, wherein each area has an index block format in which index data for each tile area is collected. The index data includes header data in which an area formed by dividing the hierarchical structure is associated with the index block corresponding to the area.
The drawing unit searches the header data based on information on a drawing target area, specifies an area obtained by dividing the hierarchical structure including the drawing target area, and the index associated with the area. The image processing apparatus according to claim 6, wherein data is read in units of the tile data from the hierarchical color buffer and the hierarchical Z buffer by referring to a block.   The image processing according to claim 2, wherein the hierarchical color buffer and the hierarchical Z buffer share one index data in which the tile area is associated with both tile data. apparatus.   The image processing apparatus according to claim 1, wherein the hierarchical data update unit is connected to the hierarchical data storage unit independently of the drawing unit.   An address conversion table in which consecutive addresses given to tile data read at one time are associated with addresses assigned to each tile data when the drawing unit reads tile data from the hierarchical color buffer and the hierarchical Z buffer. 10. The memory device according to claim 2, further comprising: a memory controller configured to convert addresses designated when the drawing unit reads data from the hierarchical color buffer and the hierarchical Z buffer into the continuous addresses. An image processing apparatus according to any one of the above. A step in which a main processor of the image processing device executes a program and issues an image processing request including information for drawing an image to be newly displayed on the display device ;
The graphic processor of the image processing device receives the image processing request, executes image processing, is stored in a memory, and is used for display in the display device, and holds color information for each pixel of the image to be drawn, and is a color value pixel plane And updating a hierarchical color buffer and a hierarchical Z buffer in which a Z-value pixel plane that holds depth information from the viewpoint for each pixel is hierarchized in correspondence with a plurality of resolutions of the drawing target image , and
The step of updating the hierarchical color buffer and the hierarchical Z buffer comprises:
Based on the image processing request, in the hierarchical color buffer and the hierarchical Z buffer, the hierarchy corresponding to the resolution and area of the image to be newly displayed and the data in the area are respectively set as the drawing target hierarchy and area. read from the specific to the memory the data, and writing the interior of the drawing color buffer and rendering Z buffer,
Performing an image processing operation based on the image processing request and updating data of the drawing color buffer and the drawing Z buffer;
Update the data of the original hierarchy and area in the hierarchical color buffer and the hierarchical Z buffer stored in the memory with the updated data in the drawing color buffer and the drawing Z buffer, and after the update Updating the data in the corresponding area of the other hierarchy by scaling the data of
An image processing method comprising: A function for executing an image processing request including information for drawing a new image to be displayed;
A color value pixel plane that holds color information for each pixel of an image to be drawn and a Z value pixel plane that holds depth information from the viewpoint for each pixel are layered according to the multiple resolutions of the drawing target image A function of storing the hierarchical color buffer and hierarchical Z buffer in a memory;
A computer program that accepts the image processing request, executes image processing, and causes the computer to realize the function of updating the hierarchical color buffer and the hierarchical Z buffer,
The function of updating the hierarchical color buffer and the hierarchical Z buffer is:
Based on the image processing request, in the hierarchical color buffer and the hierarchical Z buffer, the hierarchy corresponding to the resolution and area of the image to be newly displayed and the data in the area are respectively set as the drawing target hierarchy and area. It reads the data specified by a function of writing the drawing color buffer and rendering Z buffer,
A function of performing image processing calculation based on the image processing request and updating data of the drawing color buffer and the drawing Z buffer;
The data of the original hierarchy and area in the hierarchical color buffer and the hierarchical Z buffer are updated with the updated data in the drawing color buffer and the drawing Z buffer, and the updated data is subjected to scaling processing. And the function to update the data in the corresponding area of other layers ,
A computer program for causing a computer to realize the above. A function for executing an image processing request including information for drawing a new image to be displayed;
A color value pixel plane that holds color information for each pixel of an image to be drawn and a Z value pixel plane that holds depth information from the viewpoint for each pixel are layered according to the multiple resolutions of the drawing target image A function of storing the hierarchical color buffer and hierarchical Z buffer in a memory;
A recording medium storing a computer program, wherein the computer implements a function of accepting the image processing request, executing image processing, and updating the hierarchical color buffer and the hierarchical Z buffer,
The function of updating the hierarchical color buffer and the hierarchical Z buffer is:
Based on the image processing request, in the hierarchical color buffer and the hierarchical Z buffer, the hierarchy corresponding to the resolution and area of the image to be newly displayed and the data in the area are respectively set as the drawing target hierarchy and area. It reads the data specified by a function of writing the drawing color buffer and rendering Z buffer,
A function of performing image processing calculation based on the image processing request and updating data of the drawing color buffer and the drawing Z buffer;
The data of the original hierarchy and area in the hierarchical color buffer and the hierarchical Z buffer are updated with the updated data in the drawing color buffer and the drawing Z buffer, and the updated data is subjected to scaling processing. And the function to update the data in the corresponding area of other layers ,
The recording medium which recorded the computer program characterized by making a computer implement | achieve.

Images