JP2008129793A - Document processing system, apparatus and method, and recording medium with program recorded thereon - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make it possible to highly accurately retrieve a table in a document or a part of the table by extracting a text including a language expression similar to the element name of a line or a column of an emphasized portion in emphasizing a partial item in the table, and utilizing the extracted text for retrieval. <P>SOLUTION: The element name of a line or a column of a table is extracted, an item emphasized in the table is extracted, and meta data relative to the table are extracted on the basis of the element name of the line or the column relative to the item of the table. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention provides a system and method for retrieving and utilizing desired images and tables contained in a document by extracting images and tables included in the document and metadata related to the images and tables. And a recording medium on which the program is recorded.

  With the widespread use of electronic documents, there is an increasing demand for effective use of them. Operations on electronic documents are typically created and used, then stored and saved, and then edited and processed to create new documents and reuse them to reduce document creation costs. is there. Similarly, there is a demand for taking a printed document into a computer and reusing the contents. In order to efficiently reuse a printed document and an electronic document, a search technique for searching for necessary information from a large amount of documents is important. The document includes not only character information but also image information such as diagrams, tables, and photographs, audio information, and the like, and is considered to be particularly frequently used. In the case of character information contained in a document, it can be easily searched by matching the specified search word with character information. However, in the case of image information, etc., it does not have character information itself. A technique for adding metadata for search to image information or the like has been proposed.

  Japanese Patent Laid-Open No. 11-025113 divides a document including an image into an image region and a character region, and extracts text describing the contents of the image from the character region.

  For example, as the text describing the contents of the image, the text including the caption and the image number word included in the caption is extracted from the character area, and is associated with the image closest to the caption.

  In this technique, the text and the image are associated with each other mainly based on the pixel unit distance between the regions. However, the relevance between the text and the image cannot be determined only by the spatial distance. In this technique, the text and the image are associated with each other using a word related to the image (image number word or image instruction word). However, the relevance between the text and the image cannot be determined only by including the word.

  As techniques for analyzing a document image and extracting and structuring its logical structure, for example, the following methods have been proposed.

  Japanese Patent Laid-Open No. 11-250041 extracts areas such as text, images, and separators and their layout structure from a page image of a document obtained by scanning a printed document, and further extracts logic such as a title, header, and body from the text area. Extract objects. Also, the reading order and the relationship with other logical objects are determined for the logical objects in the page, and the logical structure in page units is extracted.

However, with this technique, only the logical structure of the document is extracted, and attention is paid to the image area included in the document and the relation with the text area is not performed.
JP-A-11-025113 Japanese Patent Laid-Open No. 11-250041 JP-A-9-44594 Japanese Patent Laid-Open No. 11-073474 JP-A-6-96275

  In order to perform a more accurate search for images, tables, etc. in a document, it is important how to extract information relating to images, tables, etc. However, conventionally, there is only a general method for extracting the text caption itself and the text in the nearby text as metadata for search based on one caption attached to the image, table, etc. There was no metadata extraction method that considered the structure and relationships of tables.

  The present invention has been made in view of the above problems, and an object of the present invention is to enable a more accurate search for images and tables by more accurately grasping the relationship between images, tables and texts in a document. And In particular, if some parts in the table are highlighted by coloring or decorating the background or characters, the element names of the highlighted parts in rows and columns are selected from the nearby text. An object of the present invention is to make it possible to easily and accurately search a table or a part of a table in a document by searching for texts containing similar language expressions, extracting them as metadata, and using them when searching.

  In addition, the present invention has been made in view of the above problems, and when searching for text including a linguistic expression similar to an element name of a row or column of a table and extracting it as metadata from nearby text, Gives high importance to texts that contain linguistic expressions similar to the highlighted part row or column element names, and can be used during search to easily and accurately use a table or part of a table in a document The purpose is to be able to search.

  Documents such as images and tables in documents in a document processing system that can retrieve and use desired images and tables in documents by extracting the images and tables in the documents and metadata related to the images and tables. In order to search for components, the layout of the document to be searched is analyzed in advance, and the components such as images and tables included in the document, as well as table items, are analyzed. Means for extracting information to be characterized; metadata extraction means for extracting metadata relating to a table included in the document based on feature information of items in the table in the document; a table included in the document; and the metadata Metadata storage means for associating and storing metadata extracted by the extraction means, and search condition input for inputting a search condition for searching a table included in the document And a search means for searching for metadata associated with the table based on the input search condition, and a table or a part of the table in the document searched by the search means is displayed as a search result. And a search result display means.

  As described above, according to the present invention, metadata related to images and tables included in a document image can be extracted with high accuracy.

  Further, according to the present invention, by using the extracted metadata, it becomes possible to easily and accurately search for an image or a table included in a document image according to a user request.

  Further, according to the present invention, as a result, images and tables in the document can be efficiently reused.

  Further, according to the present invention, by using the extracted metadata, it is possible to efficiently classify, organize, and manage documents and images and tables in the documents.

  In this embodiment, at the time of document registration, a document image is analyzed, areas such as characters, diagrams, and tables and their layouts are extracted, and based on this, logical attributes of the areas and logical relationships that structure their logical relationships are extracted. The structure is extracted, and the search metadata is extracted based on it. However, it is not always necessary to analyze the logical structure of the document, and if the information necessary to extract the search metadata is obtained Any analysis may be performed.

  Preferred embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 is a block diagram showing a basic configuration of a computer apparatus in which a document processing system according to an embodiment of the present invention is constructed.

  In FIG. 1, reference numeral 101 denotes a CPU which executes various controls in the document processing apparatus according to the present embodiment. The content of the work is instructed by a program on the ROM 102 or RAM 103 described later. Further, a plurality of computer programs can be operated in parallel by the function of the CPU itself and the computer program mechanism. Reference numeral 102 denotes a ROM which stores a computer program area and a data area in which a control procedure by the CPU is stored. Reference numeral 103 denotes a RAM which stores a control program for processing by the CPU 101 and provides a work area when the CPU 101 executes various controls. Reference numeral 104 denotes a keyboard having various function keys such as alphabet key, hiragana key, katakana key, character symbol input key such as a punctuation mark, and cursor movement key for instructing cursor movement. Provide an environment. A pointing device such as a mouse can also be included. Reference numeral 105 denotes an address bus or a data bus connected to each component. 106 is an external storage device for storing various data and the like, and drives a recording medium such as a hard disk, a flexible disk, an optical disk, a magnetic disk, a magneto-optical disk, a magnetic tape, and a nonvolatile memory card, and a storage medium. And a drive for recording information. The stored computer program and data are called completely or partially on the RAM when necessary by an instruction from a keyboard or the like or an instruction from various computer programs.

  Reference numeral 107 denotes a display, which includes a display or the like, and displays various input operation states to the user.

  Reference numeral 108 denotes a communication device for communicating with other computer devices and the like, which communicates with a device located in a remote place (not shown) via a network (LAN), etc. Can be shared.

  Reference numeral 109 denotes an image scanner for reading an image, which optically reads a set paper document one by one and converts an image signal into a digital signal sequence. The read image data can be stored in an external storage device, RAM, or the like.

  A storage medium such as a ROM or a RAM stores a computer program or data for realizing the data management of the present embodiment, and functions by the CPU reading and executing the program code stored in the recording medium. However, the type of storage medium is not limited. Also, an external recording device storing the program and data according to the present invention is supplied to the system or the device, and the program is copied from the external storage device onto a rewritable storage medium such as a RAM. The external storage device may be anything such as a flexible disk, a CD-ROM, a hard disk, a memory card, and a magneto-optical disk.

  Reference numeral 104 denotes a keyboard or mouse that provides various input operation environments by the user, but any device such as a touch panel or a stylus pen may be used as long as it provides various input operation environments from the user.

  The communication means 108 may be anything such as RS232C, USB, IEEE 1394, P1284, SCSI, modem, wired communication, infrared communication, wireless communication such as IEEE802.11b, etc., and devices connected to the apparatus of the present invention also communicate. It doesn't matter as long as it has means.

  In addition, the image data may be input via an input device such as a network scanner or a copying apparatus connected to the communication device 108 (not shown) instead of the 109 image scanner. Data may also be stored in an external storage device such as a server or a copier connected by a communication device, instead of an external storage device or RAM.

  The configuration described above is an example in the present embodiment, and the present invention is not particularly limited to this.

  The operation at the time of document registration in this embodiment will be described in detail. FIG. 3 is a flowchart showing an example of the flow of document registration processing. This processing program is stored in the ROM 102 and executed by the CPU 101.

  In the description of the operation of the present embodiment, as an example, a description will be given for a paper document read using an input device such as the image scanner 109. However, not only the paper document but also an editing software such as a word processor was used. It may be a document, a document described in HTML or the like, or an electronic document in a format such as PDF. However, in the case of an electronic document, processing such as format conversion is required in the input processing in step S301. Hereinafter, it demonstrates in detail using FIG.

  In step S301, an input device such as an image scanner 109 or a copier connected to a network is used to read a document composed of one or a plurality of pages in which images and character information are mixed, and digitized page units. This is a step of obtaining a document image. Document images input by the input device include binary images and color images. Note that after obtaining the digitized document image, noise removal processing, orientation and inclination correction processing, and the like may be performed on each page image. As a method for determining and correcting the orientation and inclination of the page image, for example, there are methods as disclosed in JP-A-9-44594, JP-A-11-073474, and the like. It doesn't matter.

  Step S302 is a step of performing processing for analyzing the document image input in step S301. The process of step S302 will be described later with reference to FIG.

  In step S303, the logical structure of the document is extracted by performing analysis based on various information about each area, layout extraction results, and characteristics of character information included in the character area. The logical structure is obtained by extracting and assigning logical logical attributes to the area or page extracted in step S302 and estimating and structuring the logical relationship between them. The process of step S303 will be described later with reference to FIG.

  Next, in step S304, based on the logical structure extracted in step S303, processing for extracting search metadata related to images and tables from the character area in the page image is performed. The process of step S304 will be described later with reference to FIG.

  The document registration process described with reference to FIG. 3 is an example of the process according to the present invention, and various other processes are conceivable. The order of processing and the processing content may not be as described above.

  FIG. 4 is a flowchart showing an example of the processing in step S302 of FIG. 3, and is a flowchart showing an example of processing at the time of document image analysis in the present invention. This processing program is stored in the ROM 102 and executed by the CPU 101.

  If the document input in S301 of FIG. 3 holds a sentence as a character code, such as an electronic document, the character recognition process in step S402 is not necessary. Hereinafter, it demonstrates in detail using FIG.

  In step S401, area division is performed for each page image of the document read in step S301 in FIG. 3, and a rectangular area containing images such as characters, diagrams, tables, and photographs is classified into the type and size of the rectangle, and within the page. It is extracted together with physical information such as position coordinates at. For the character area, the reading direction and the character size of the character string are detected, and the character string line and the character are extracted based on the detected direction. Here, an area in which the direction of the character string is the same and the character size, the inter-character value, and the inter-line value are almost uniform is extracted as a set of character areas. For non-character areas, photographs, tables, frames, lines, etc. are detected and extracted as areas. Further, physical information based on the features of the region may be extracted at the time of region extraction. For example, in the case of a character area, information relating to character decoration, character style, etc. may be extracted together. In the case of a non-character area, information obtained by analyzing image features may be extracted. . When the input document image is a multi-value such as a color image, the region division process can be performed in the same manner by converting it to binary. As a region dividing method in this step, for example, the method disclosed in Japanese Patent Laid-Open No. 6-96275 can be used. However, the method is not limited to this method, and any method can be used as long as regions such as characters and images can be extracted. It doesn't matter.

  FIG. 9 shows an example of various physical information about a page image and a region extracted from each page image in a certain document.

  Next, the process proceeds to step S402, and character recognition processing is performed on all the character areas extracted in step S401.

  Next, in step S403, the layout of each page of the document is extracted, and the spatial relationship between the rectangular areas in each page image is extracted according to the physical information of the rectangular areas. For example, as a spatial relationship between two areas in the page image, a state in which the areas exist, up and down, left and right directions, two areas are overlapped, touched, included, etc. The size relationship between the areas is calculated and determined using the position coordinates and size of each rectangular area. The above analysis results can be expressed in a tree structure or network structure for each page. The relationship between the rectangular regions and the method of expressing them are just examples in the present embodiment, and other relationships may be extracted, and the analysis results may be expressed by other methods. Absent. For example, the relative position and size of each rectangular area with respect to the entire page may be extracted as the layout.

  FIG. 10 shows an example of the result of extracting the spatial relationship between regions in a page image.

  All processing results from step S401 to step S404 are stored in a storage medium such as a RAM or an external storage device.

  The processing from step SS401 to step S403 is performed for all page images of the input document.

  The document image analysis process described above with reference to FIG. 4 is an example of the process in the present invention, and various other processes are conceivable. The processing order and the processing content may not be as described above. .

  FIG. 5 is a flowchart showing an example of the process in step S303 of FIG. 3, and is a flowchart showing an example of the logical structure analysis process in the present invention. This processing program is stored in the ROM 102 and executed by the CPU 101. Hereinafter, it demonstrates in detail using FIG.

In step S501, processing for determining the reading order of pages and areas is performed.
The reading order of pages and areas can be determined by the reading direction and positional relationship of character areas, the grouping direction that can be determined from the reading directions of main character areas in a document, and the like. For example, in a page written horizontally in Japanese, the reading order of character areas is given from the top to the bottom of the same row and from the left to the right. This is an example of a determination method, and there are various other methods.

  The reading order determination process in step S501 is not necessarily required, and may be omitted depending on the type of document.

  In step S502, a process of analyzing the logical attribute of the page or area and the relationship between the areas is performed.

  For example, a page that is at the top of a document and has more blank space than other pages and has a character region that is characteristic compared to other characters in the document can be estimated as a cover page in the document.

  In addition, a character area that is low in common / regularity with other character areas in the document and in the arrangement and character, and a character area that is characteristic of a character, is a title if it is at the top of the document excluding the header. If it appears at a position, it can be estimated that it is a headline. In addition, the character area closest to the image area such as a photograph, a figure, or a table can be estimated as a caption for the image area, and a relationship with the image area can be established.

  These logical attributes and logical relationships can be estimated for characteristic pages and areas, and cannot be assigned to all pages and areas. In addition, for a page or area, it is not always possible to determine a single logical attribute or logical relationship, so multiple potential logical attributes or relationships may be added, indicating the strength of that possibility. A numerical value or the like may be given.

  Further, a numerical value indicating the strength of the relationship may be given to the logical relationship.

  The logical attributes and relationships listed here and their analysis rules are examples, and there are many others.

  In step S503, it is determined whether the analysis target area is a table area. If it is determined as a table, the process proceeds to step S504.

  Step S504 is a step of analyzing the table region using the physical information of the region and the spatial relationship between the regions, and extracting attributes indicating the structure and features of the table. Here, element names of rows and columns, and attributes indicating emphasis of areas and item values for each item are extracted. For example, row and column element name portions can be identified using the position in the table, the type of ruled line, etc., and linguistic features. Further, for example, with respect to the area indicating the table element, it can be determined whether or not the area is emphasized depending on whether or not an image area indicating the background color or decoration is included. As for the item values in the table, it is determined whether or not emphasis is applied based on the character size, character color, character style, etc. different from the characters of the other item values in the table.

  The example of FIG. 8 indicates that the item values in the first row and first column of the table 806 are the element names of the rows and columns in the table. In addition, the background of the element in the 3rd row × 5th column is a halftone dot, which indicates that this element is emphasized more than other elements. In the example of FIGS. 9 and 10, 2 of area 6 indicating the table element in the 3rd row × 5th column includes 1 of 2 of image area 6 of the texture type, and further 1 of 2 of area 6 Includes 2 in the character area 6. The other table elements do not have the same image area but include a character area. From this, it can be estimated that 2 of the table element region 6 is emphasized more than other table regions.

  The processing results from step S501 to step S504 are all stored in a storage medium such as a RAM or an external storage device.

  FIG. 11 shows an example of the result of analyzing the logical structure of a document.

  The logical structure analysis process described above with reference to FIG. 5 is an example of the process in the present invention, and various other processes are conceivable, and the processing order and processing contents may not be as described above. . For example, the content to be extracted as the logical structure may not be a semantic attribute of the page or area, or other relationships may be extracted.

  FIG. 6 is a flowchart showing an example of the process in step S304 of FIG. 3, and shows the search metadata extraction process in this embodiment. This processing program is stored in the ROM 102 and executed by the CPU 101. Hereinafter, it demonstrates in detail using FIG.

  First, in step S601, a search target area is extracted from the logical structure analysis result of the document. Here, areas such as photographs, diagrams, and tables are set as search targets, but other areas may be extracted as search targets.

  Next, in step S602, text information serving as search metadata for the area extracted in step S601 is extracted. First, an area from which metadata is extracted is acquired. Here, the character area that is logically connected to the image area to be searched and the character area that is logically connected to the character area are acquired, but the other areas are extracted as metadata. It may be a target. Next, the text portion describing the image area is extracted from the text of the acquired character area.

  For example, an image number character string (such as “Table 1”) and an image name character string (such as “product specification comparison”) are extracted from text in a character area whose logical attribute is “caption”, and each is used as metadata. Alternatively, a sentence including an image number character string may be extracted from a character area having a logical attribute “paragraph” around the image area to be searched. Also, a word indicating the direction of the image such as “upper table” and a sentence including the word indicating the image are extracted, and the direction of the image indicated by the word and each page in the page extracted in step S404 are extracted. By collating the spatial relationship of areas, correlating images with metadata, extracting sentences that include words that indicate the position of the page and words that indicate the image, such as “Next (Figure) Page” The page position indicated by the word and the page structure of the document can be collated to associate the image with the metadata.

  Next, in step S603, it is determined whether or not the object extracted as a search target in step S601 is a table area. Here, in the case of a table area, the process proceeds to step S604.

  In step S604, a process is performed for determining the weights for the element names of the rows and columns for the table area extracted as the search target in step S601. The weighting can be performed by, for example, the background of the area indicating the item value in the table, the presence or absence of character emphasis, and the like. Further, each word included in the element name may be weighted using information such as the part of speech or meaning of the word included in the element name, or the related word of each word included in the element name may be weighted. It may be given. Moreover, you may determine a weight using other information.

  For example, in the example of FIG. 8, 2 in the region 6 indicating the table element in the 3rd row × 5th column is emphasized in the table, so that “B” which is the element name in the 3rd row of the table and the 5th column The “output speed” that is the element name of the eye is set to have a higher weight than the other element names.

  In step S605, a text portion related to the element name of the table is extracted as search metadata from the text of the character area to be extracted in step S602. For example, a sentence including each word included in the element name of the table is extracted from a character area having a logical attribute “paragraph” around the table area to be searched. At this time, the position of the element name used for extraction in the table may be given together as information.

  In the example of FIG. 8, the sentence “The size of A is ...” and the sentence “The output speed of B is the fastest among the products” are extracted.

  In step S606, an importance level is calculated for the text extracted in step S605 according to the weights of the element names of the rows and columns determined in step S604.

  For example, in the example of FIG. 8, the element name “B” in the third row of the table and the element name “output speed” in the fifth column are set higher in weight than other element names. The sentence “The output speed of B is the fastest among the products” is more important than the sentence “A size is ...” extracted in step S605.

  The metadata extraction methods described in step S602 and step S605 are examples, and various other metadata extraction methods can be considered. In addition, when metadata is extracted, information indicating the reason for extraction as metadata may be output, and the relationship between the image and metadata is strengthened according to the logical attributes and logical relationships of the extraction target area. A numerical value indicating the length may be output. Here, the text included in the character area is extracted as metadata, but information other than text may be extracted as metadata.

  In step S607, the search metadata extracted in steps S602 and S605 and the search target area are associated with each other and stored in the DB.

  FIG. 2 shows an example of the data structure when the image area to be searched and the extracted metadata are stored in association with each other.

  Step S608 is a step of determining whether or not the next search target area exists. If there is a search target, the process returns to step S602, and the processes from step S602 to step S607 are all performed in step S601. This is done for the area.

  The processing flow of FIG. 6 is an example of the metadata extraction processing in the present embodiment, and the processing order and processing content may not be as described above.

  FIG. 8 shows an example of a result of performing region extraction processing on a certain page image.

  In FIG. 8, reference numeral 800 denotes a scanned page image, and reference numerals 801 to 807 denote extracted object regions. 801 to 805 are character areas, but the direction of the character string is the same, the character size, the inter-character value and the inter-line value are almost uniform, and the arrangement in the line direction (indentation, centering, alignment, etc.) is the same Are extracted as one character area. Reference numeral 806 denotes a table area, and reference numeral 807 denotes one of the items in the table.

  These drawings show an example of the region extraction result in this embodiment, but other region extraction results may be used as long as the image and character regions can be extracted.

  In FIG. 8, a document image in which an image and text are mixed is taken as an example. However, a plurality of types of objects are not necessarily mixed, and for example, a document image composed only of images may be used.

  FIG. 9 shows an example of various physical information about a page image and a region extracted from each page image in a certain document.

  In FIG. 9, physical information such as the page size, the resolution at the time of reading, and the storage position of the digitized page image data is given to the page image. In addition, physical information such as a region type such as a character region and an image region, a size of the rectangular region, and a position coordinate in the page is assigned to each extracted rectangular region. Further, the character area is assigned character size and character information as a result of character recognition, and the image area is assigned image type information such as a photograph, a diagram, a table, and a symbol image. For example, the page image 1 has a width of 290 mm, a height of 210 mm, and a processing resolution of 300 dpi. A region 1 is a character region having a width of 40 mm and a height of 12.5 mm at a position of an X coordinate of 10 mm and a Y coordinate of 10 mm, and indicates that a character string is described with a character size of 8 points. FIG. 9 shows an example of the physical information of the area in this embodiment. However, the physical information is not limited to this, and if the next step layout can be extracted, other information is extracted. May be. For example, in FIG. 9, the size and position coordinate information of the rectangular area are extracted, but the upper left position coordinates and the lower right position coordinates of the rectangular area may be extracted.

  FIG. 10 shows an example of the result of extracting the spatial relationship between regions in a page image.

  In FIG. 10, as a spatial relationship with respect to two areas in the page image, a state in which the areas are in the vertical and horizontal directions, the two areas are overlapped, in contact with each other, included, Further, when the two areas are not in contact with each other, the relative distance between the two adjacent areas is expressed by a network structure. For example, the spatial relationship between region 1 and region 2 indicates that region 2 is below region 1. The spatial relationship between the region 6 and the region 6 indicates that the region 6 includes 1 in the region 6. FIG. 10 shows an example of the spatial relationship of the areas in the present embodiment. However, the spatial relationship is not limited to this, and if the logical structure analysis of the next step can be performed, other information is extracted. May be.

  FIG. 11 shows an example of the result of analyzing the logical structure of the document, and shows the logical attributes of the extracted pages and areas, their reading order, logical relationship, and the like. This is one example of the result of analyzing the logical structure, but the analysis result of the logical structure may be in any format and content as long as the metadata extraction rules can be applied. Things can be considered.

  In FIG. 11, page images 1, 2,... Are logical information related to each page image of the read document, and are given logical attributes and the like for the page. The areas 1, 2,... Are logical information related to the areas extracted from the page image 1, and are given logical attributes and the like for the areas. Solid arrows connecting pages and regions indicate the reading order and logical relationship of the pages and regions, and dotted lines without arrows indicate inclusion relationships. For example, page image 1 includes region 1, region 2,..., Of which region 3 has logical attribute “headline”, logical attribute “paragraph” region 4, and logical attribute “ The reading order is given in the order of “caption” area 5. The area 5 whose logical attribute is “caption” and the area 6 whose logical attribute is “table” are connected by “relationship between table and caption”. The logical attribute “1” in the area 6 with “table element name”, the logical attribute “2” in the area 6 with “table item”,... Are included in the area 6 with the logical attribute “table”. 20 in the region 6 that is the “table item value” is included in 2 of the region 6.

  FIG. 2 shows an example of the data structure when the area to be searched and the extracted metadata are stored in association with each other.

  FIG. 2 shows that text or the like extracted from a document is assigned as metadata to an image or table included in the page image 1. Here, as an example, for a certain table, the value indicating the reason why the character string “Product Specification Comparison” etc. was extracted as metadata as the text explaining the table, and the strength of the relationship between the table and the text It is given with.

  In the present embodiment, in particular, the text “A size is ...” and “B output speed is the fastest among the products” extracted from the surrounding character area using the element names in the table. These are given as metadata together with importance “2” and importance “3”, respectively. Although not shown in the drawing, the position of the element name used for extraction in the table may be added as information to the metadata extracted using the element name of the table.

  This is an example, and the stored metadata and storage method are not limited to this. These are stored in a recording medium such as a storage device such as DISK, ROM, RAM, etc. in FIG.

  FIG. 7 is a flowchart showing an example of processing for searching for an image, a table, or the like included in a document in the present invention. This processing program is stored in the ROM 102 and executed by the CPU 101. Hereinafter, this will be described in detail with reference to FIG.

  In this embodiment, a search is performed using search metadata associated with an image, a table, or the like as shown in FIG. The search is given by the user search conditions such as a search keyword, a list of keywords, a natural sentence, etc., and the search conditions are compared with metadata associated with each image, table, etc., and the search conditions are met. Pick up images and tables to which data is assigned and display them as search results.

  Step S1301 is a part for manipulating data on the RAM in FIG. 1 so that search index information and the like can be used. The data to be searched, its metadata, etc. can be held on a computer such as a PC on the LAN via the NCU of FIG. 1 or on a computer on the external network via the DTU.

  Step S1302 shows a process in which the user inputs a sentence or a word as a search condition. Here, the user can input a natural sentence expressing an image, a table, or the like to be searched from the KB of FIG. 1 as a search condition. Further, for example, this search condition can be received by the PC in FIG. 1 and input via the LAN.

  In step S1303, a sentence or word input as a search condition is analyzed using a natural language processing technique, such as morphological analysis or syntax analysis. Various known techniques can be used for each of these analysis techniques and methods, but here the sentence is divided into a collection of semantic words, and the part of speech and meaning of the word that is formed by the division. Anything can be used as long as it can extract information and information on the relationship between those words in the sentence.

  In step S1304, the analysis result obtained in step S1303 searches for images, tables, and the like included in the search metadata shown in FIG.

  In step S1305, it is determined whether there is a candidate for the image or table searched in step S1304. If no candidate is found, the search ends with no corresponding candidate. If there is even one candidate, the process proceeds to step S1306.

  In step S1306, for the image or table that is the candidate searched in step S1304, the relationship between the words in the search condition and the relationship between the metadata including the relationship are compared, so that a structure similar to the word relationship is obtained. Perform a process to check if it exists.

  In step S1307, processing for presenting to the user, as a search result, an image whose structural similarity is found in step S1306 from the candidates searched in step S1304.

  At that time, based on the similarity of the structure found in step S1306, the importance assigned to the metadata, etc., the similarity between the search condition and the metadata of each image or table picked up is obtained, and the similarity is high. The search results may be presented in order. The degree of similarity referred to here indicates how appropriate the search condition input by the user is as an expression indicating the relationship with the metadata assigned to each image, table, or the like.

  At that time, a collection of a plurality of images and tables may be used as the search result, or a part of the table corresponding to the search condition may be used as the search result.

  For example, when the search condition “product with fast output speed” is given by the user, this example sentence is broken down into words such as “output speed”, “fast”, “ga”, etc. And part-of-speech information such as “connectives”. Each word can be analyzed to have a relationship such as “modification”. Next, in the analysis result, an image or table in which metadata such as “output speed” and “fast”, which are important words having parts of speech such as nouns, is included, that is, in FIG. Get as a candidate. Next, for these candidates, the relationship between the words of the analysis results and the relationship between the metadata including them are compared. In the example of the search condition, “output speed” and “fast”, “fast” and “product” are connected in a modification relationship. The metadata by the element name of the table assigned to the area 6 in FIG. 2 includes “output speed” and “fast”, which are linked in a modification relationship. From this, it can be seen that a structure similar to the example sentence can be found in the metadata of the table in the region 6.

  For example, when a search condition “output speed of B” is given by the user, this example sentence is decomposed into words “B”, “no”, and “output speed”, and “B” and “ It is obtained as an analysis result that the “output speed” is in a modification relationship. Next, the table of the region 6 in FIG. 2 is acquired as a corresponding candidate in which “B” and “output speed” in the analysis result are included in the metadata. The metadata by the element name of the table assigned to the area 6 includes “B” and “output speed”, which are linked by a modification relationship. Therefore, in the metadata of the table of the area 6, It can be seen that a structure similar to the example sentence can be found. In this case, since “B” and “Output speed” included in the search conditions are the element names of the table, the table items (values) pointed to by these element names can be used as search results. Good.

  This is an example of a search method, and the search method is not limited to this, and any method may be used.

  In addition, by using the search result, an image, a table, or the like included in the document can be efficiently reused. For example, by editing and processing data obtained by selecting a desired image or table from a list of search results using editing software such as a word processor, or inserting it into another document Can be reused. This is an example of a reuse method, and is not limited to this, and any method may be used.

  Also, by using the metadata, it is possible to efficiently classify, organize, and manage documents and images and tables in the documents. For example, it is possible to analyze words given as metadata and classify images and tables by related categories, etc. The categories to be classified may be given by the user, or statistical methods such as clustering May be automatically classified according to Further, at the time of classification, the similarity between the category and metadata such as images and tables may be calculated and used for classification. This is an example of a classification method and an example of a document management method. The document management method is not limited to this, and any method may be used.

(Other embodiments)
In the present embodiment, the analysis results and analysis rules shown in FIGS. 2, 9, 10, and 11 are stored in the storage device such as the external storage device, ROM, or RAM in FIG. Can be obtained from a LAN or the like and stored in a storage device.

  In addition, the present invention can be applied to a system composed of a plurality of devices (for example, a host computer, interface device, reader, printer, etc.), but can also be a device composed of a single device (for example, a copier, a facsimile machine, etc.). May be adapted.

  In addition, an object of the present invention is to supply a storage medium (or recording medium) in which software program codes for realizing the functions of the above-described embodiments are recorded to a system or apparatus, and a computer (or CPU or CPU) of the system or apparatus. Needless to say, this can also be achieved by the MPU) reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that some or all of the actual processing is performed, and the functions of the above-described embodiments are realized by the processing.

  Further, after the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

1 is a block diagram showing a basic configuration example of a computer device in which a document processing system according to the present invention is constructed. The figure which shows the example of the data structure which links | relates and stores the image extracted from the document image, and metadata The flowchart which shows the example of the document registration process in this invention FIG. 4 is a detailed diagram of the processing in step S302 in FIG. 3, and is a flowchart illustrating an example of processing at the time of document image analysis; FIG. 4 is a detailed diagram of the processing in step S303 in FIG. 3, and is a flowchart illustrating an example of logical structure extraction processing; FIG. 4 is a detailed diagram of the processing in step S304 in FIG. 3, and is a flowchart illustrating an example of search metadata extraction processing in the first embodiment Flowchart illustrating an example of processing for searching for an image included in a document Example of result of region extraction processing for page image Diagram showing examples of various physical information about page images and areas The figure which shows the example of the result of having extracted the spatial relationship of each area | region Diagram showing an example of the result of analyzing the logical structure of a document

Claims (43)

In a document processing system that can search and use images and tables in a document by extracting images and tables in the document and metadata related to the images and tables,
In order to search for document components such as images and tables in the document, means for analyzing in advance the document to be searched and extracting information characterizing each component included in the document When,
Metadata extraction means for extracting metadata relating to a table included in the document based on feature information of items in the table in the document;
Metadata storage means for associating and storing the table included in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table included in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing system comprising: In a document processing system that can search and use images and tables in a document by extracting images and tables in the document and metadata related to the images and tables,
In order to search for document components such as images and tables in the document, means for analyzing in advance the document to be searched and extracting information characterizing each component included in the document When,
Metadata extraction means for extracting metadata relating to the table included in the document based on element names of rows and columns relating to items of the table in the document;
Metadata storage means for associating and storing the table in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing system comprising: In a document processing system that can search and use images and tables contained in documents by extracting the images and tables in the document and the metadata related to the images and tables,
A document input means for inputting a document in which images, tables, and sentences are mixed;
An area extraction means for recognizing and extracting an image or table in the document and an area including a sentence from the document input via the document input means;
Layout analysis means for analyzing a layout related to the area extracted by the area extraction means in each page of the document input via the document input means;
Logical structure analysis means for analyzing the logical structure of the document based on the layout extracted by the layout analysis means for each page of the document input via the document input means;
Metadata extraction means for extracting metadata relating to the table included in the document based on the element names of the rows and columns relating to the items of the table in the document based on the logical structure analyzed by the logical structure analysis means; ,
Metadata storage means for associating and storing the table included in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing system comprising: The document processing system according to claim 2, wherein:
The document processing system, wherein the metadata extraction means extracts metadata about a table included in the document based on element names of rows and columns related to emphasized items in the table. The document processing system according to any one of claims 1 to 3,
Metadata importance calculation means for calculating importance for metadata extracted by the metadata extraction means;
With
The metadata storage means stores the metadata extracted by the metadata extraction means, the importance calculated by the metadata importance calculation means, and a table included in the document in association with each other,
The document processing system, wherein the search means calculates a similarity between a search condition and metadata based on metadata and importance associated with the table. The document processing system according to claim 5,
Element name weighting means for weighting the element names of the rows and columns of the table;
With
The metadata extraction means extracts metadata related to a table included in the document based on element names of rows and columns of the table,
The document processing system, wherein the metadata importance calculation means calculates the importance of the metadata in accordance with an element name weight by the element name weighting means. The document processing system according to claim 6.
The element name weighting unit weights element names of table rows and columns included in the document in accordance with presence / absence of emphasis of items in the table. The document processing system according to claim 2, wherein:
The metadata extraction means extracts metadata related to a table included in the document according to a relationship between words included in element names of rows and columns of the table and a relationship between words in sentences included in the document. A document processing system. The document processing system according to any one of claims 5 to 7,
The metadata extraction means extracts metadata relating to a table included in the document according to a relationship between words included in element names of the rows and columns of the table and a relationship between words in sentences included in the document,
The metadata importance calculating means calculates the importance of the metadata according to the relationship between words included in the element names of the rows and columns of the table and the relationship between words in sentences included in the document. Feature document processing system. The document processing system according to any one of claims 1 to 9,
The search means searches for metadata associated with the table according to a relationship between words included in the search condition and a relationship between words within the search condition included in the metadata. system.   The document processing system according to claim 3, wherein the layout analysis unit extracts a spatial relationship between the regions.   4. The document processing system according to claim 3, wherein the logical structure analyzing unit extracts logical attributes of pages and areas constituting the document, determines a reading order of the areas, and extracts the logical structure. .   The document processing system according to claim 1, wherein the document is a paper document on which a book or an electronic document is printed.   The document processing system according to claim 1, wherein the document is an electronic document. In a document processing device that can search and utilize images and tables in a document by extracting images and tables in the document and metadata related to the images and tables,
In order to search for document components such as images and tables in the document, means for analyzing in advance the document to be searched and extracting information characterizing each component included in the document When,
Metadata extraction means for extracting metadata relating to a table included in the document based on feature information of items in the table in the document;
Metadata storage means for associating and storing the table included in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table included in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing apparatus comprising: In a document processing device that can search and utilize images and tables in a document by extracting images and tables in the document and metadata related to the images and tables,
In order to search for document components such as images and tables in the document, means for analyzing in advance the document to be searched and extracting information characterizing each component included in the document When,
Metadata extraction means for extracting metadata relating to the table included in the document based on element names of rows and columns relating to items of the table in the document;
Metadata storage means for associating and storing the table in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing apparatus comprising: In a document processing device that can retrieve and utilize images and tables contained in a document by extracting the images and tables in the document and metadata related to the images and tables,
A document input means for inputting a document in which images, tables, and sentences are mixed;
An area extraction means for recognizing and extracting an image or table in the document and an area including a sentence from the document input via the document input means;
Layout analysis means for analyzing a layout related to the area extracted by the area extraction means in each page of the document input via the document input means;
Logical structure analysis means for analyzing the logical structure of the document based on the layout extracted by the layout analysis means for each page of the document input via the document input means;
Metadata extracting means for extracting metadata relating to the table included in the document based on the element names of the rows and columns relating to the items of the table in the document based on the logical structure analyzed by the logical structure analyzing means; ,
Metadata storage means for associating and storing the table included in the document and the metadata extracted by the metadata extraction means;
Search condition input means for inputting a search condition for searching a table in the document;
Search means for searching for metadata associated with the table based on the input search condition;
Search result display means for displaying a table or a part of the table in the document searched by the search means as a search result;
A document processing apparatus comprising: The document processing apparatus according to any one of claims 16 to 17,
The document processing apparatus, wherein the metadata extraction unit extracts metadata about a table included in the document based on element names of rows and columns related to highlighted items in the table. The document processing apparatus according to any one of claims 15 to 17,
Metadata importance calculation means for calculating importance for metadata extracted by the metadata extraction means;
With
The metadata storage means stores the metadata extracted by the metadata extraction means, the importance calculated by the metadata importance calculation means, and a table included in the document in association with each other,
The document processing apparatus, wherein the search unit calculates a similarity between a search condition and metadata based on metadata and importance associated with the table. The document processing apparatus according to claim 19, wherein
Element name weighting means for weighting the element names of the rows and columns of the table;
With
The metadata extraction means extracts metadata related to a table included in the document based on element names of rows and columns of the table,
The document processing apparatus characterized in that the metadata importance calculation means calculates the importance of the metadata in accordance with the weight of the element name by the element name weighting means. The document processing apparatus according to claim 20, wherein
The element name weighting unit weights element names of table rows and columns included in the document according to presence / absence of emphasis of items in the table. The document processing apparatus according to any one of claims 16 to 18,
The metadata extraction means extracts metadata related to a table included in the document according to a relationship between words included in element names of rows and columns of the table and a relationship between words in sentences included in the document. A document processing apparatus characterized by the above. The document processing apparatus according to any one of claims 19 to 21,
The metadata extraction means extracts metadata relating to a table included in the document according to a relationship between words included in element names of the rows and columns of the table and a relationship between words in sentences included in the document,
The metadata importance calculating means calculates the importance of the metadata according to a relationship between words included in element names of the rows and columns of the table and a relationship between words in sentences included in the document. Feature document processing device. 24. The document processing apparatus according to claim 15, wherein:
The search means searches for metadata associated with the table according to a relationship between words included in the search condition and a relationship between words within the search condition included in the metadata. apparatus.   The document processing apparatus according to claim 17, wherein the layout analysis unit extracts a spatial relationship between the regions.   18. The document processing apparatus according to claim 17, wherein the logical structure analyzing unit extracts logical attributes of pages and areas constituting the document, determines a reading order of the areas, and extracts the logical attributes as a logical structure. .   27. The document processing apparatus according to claim 15, wherein the document is a paper document on which a book or an electronic document is printed.   27. The document processing apparatus according to claim 15, wherein the document is an electronic document. In a document processing method that can search and utilize images and tables in a document by extracting the images and tables in the document and metadata related to the images and tables,
In order to search for a component part of a document such as an image or a table in the document, a process of analyzing a document to be searched in advance and extracting information characterizing each component included in the document When,
A metadata extraction step of extracting metadata relating to a table included in the document based on feature information of items in the table in the document;
A metadata storage step of associating and storing the table included in the document and the metadata extracted by the metadata extraction step;
A search condition input step for inputting a search condition for searching a table included in the document;
A search step of searching for metadata associated with the table based on the input search condition;
A search result display step of displaying a table or a part of the table in the document searched by the search step as a search result;
A document processing method comprising: In a document processing method that can search and utilize images and tables in a document by extracting the images and tables in the document and metadata related to the images and tables,
In order to search for a component part of a document such as an image or a table in the document, a process of analyzing a document to be searched in advance and extracting information characterizing each component included in the document When,
A metadata extraction step of extracting metadata relating to a table included in the document based on element names of rows and columns relating to items of the table in the document;
A metadata storage step of associating and storing the table in the document and the metadata extracted by the metadata extraction step;
A search condition input step for inputting a search condition for searching the table in the document;
A search step of searching for metadata associated with the table based on the input search condition;
A search result display step of displaying a table or a part of the table in the document searched by the search step as a search result;
A document processing method comprising: In a document processing method that can search for and utilize images and tables contained in a document by extracting the images and tables in the document and metadata related to the images and tables,
A document input process for inputting a document in which images, tables, and sentences are mixed,
An area extraction step of recognizing and extracting an image or table in the document and an area including a sentence from the document input through the document input step;
A layout analysis step of analyzing a layout related to the region extracted by the region extraction step in each page of the document input through the document input step;
A logical structure analysis step for analyzing the logical structure of the document based on the layout extracted by the layout analysis step for each page of the document input through the document input step;
Based on the logical structure analyzed by the logical structure analysis step, based on the element names of the rows and columns related to the table items in the document, a metadata extraction step for extracting metadata about the table included in the document; ,
A metadata storage step of associating and storing the table included in the document and the metadata extracted by the metadata extraction step;
A search condition input step for inputting a search condition for searching the table in the document;
A search step of searching for metadata associated with the table based on the input search condition;
A search result display step of displaying a table or a part of the table in the document searched by the search step as a search result;
A document processing method comprising: 32. The document processing method according to claim 30, wherein:
The document processing method is characterized in that the metadata extraction step extracts metadata relating to a table included in the document based on element names of rows and columns relating to emphasized items in the table. 32. The document processing method according to claim 29, wherein:
A metadata importance calculation step for calculating the importance for the metadata extracted by the metadata extraction step;
With
The metadata storage step stores the metadata extracted by the metadata extraction step, the importance calculated by the metadata importance calculation step, and a table included in the document in association with each other,
The document processing method characterized in that the search step calculates the similarity between the search condition and the metadata based on the metadata and importance associated with the table. 34. The document processing method according to claim 33.
An element name weighting step for weighting the element names of the rows and columns of the table;
With
The metadata extraction step extracts metadata related to a table included in the document based on element names of rows and columns of the table,
The document processing method characterized in that the metadata importance calculation step calculates the importance of the metadata in accordance with an element name weight in the element name weighting step. The document processing method according to claim 34.
The element name weighting step weights element names of table rows and columns included in the document in accordance with presence / absence of emphasis of items in the table. The document processing method according to claim 30 to claim 32,
The metadata extraction step extracts metadata relating to a table included in the document according to a relationship between words included in element names of rows and columns of the table and a relationship between words in sentences included in the document. A document processing method characterized by the above. 36. The document processing method according to claim 33, wherein:
The metadata extraction step extracts metadata relating to a table included in the document according to a relationship between words included in element names of rows and columns of the table and a relationship between words in sentences included in the document,
The metadata importance calculation step calculates the importance of the metadata according to the relationship between words included in the element names of the rows and columns of the table and the relationship between words in sentences included in the document. Characteristic document processing method. The document processing method according to any one of claims 29 to 37,
The document processing characterized in that the search step searches for metadata associated with the table according to a relationship between words included in the search condition and a relationship between words within the search condition included in the metadata. Method.   32. The document processing method according to claim 31, wherein the layout analysis step extracts a spatial relationship between the regions.   32. The document processing method according to claim 31, wherein the logical structure analysis step extracts logical attributes of pages and areas constituting the document, determines a reading order of the areas, and extracts the logical attributes. .   41. The document processing method according to claim 29, wherein the document is a paper document on which a book or an electronic document is printed.   41. The document processing method according to claim 29, wherein the document is an electronic document.   15. A recording medium used for the document processing system according to claim 1.

Images