JP2016051191A - Image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing method capable of storing data, such as an article and pictures by being efficiently associated with information included in a designated part during storing the data.SOLUTION: A computer executes processing which specifies a first part in a picked-up image of a first imaging object designated by a detected feature when the feature designating a portion of the first imaging object is detected from the picked-up image obtained by imaging the first imaging object, stores information included in the specified first part in a storage part in association with first data to be stored about the picked-up image of the first imaging object, specifies a second part corresponding to the first part in a picked-up image obtained by imaging a second imaging object having a contour similar to the first imaging object, and stores information included in the specified second part in the storage part in association with second data to be stored about the picked-up image of the second imaging object.SELECTED DRAWING: Figure 9

Description

  This case relates to an image processing method.

  2. Description of the Related Art It is widely practiced to cut out articles and photographs from newspapers and books (for example, magazines and books) or copies thereof and scrap them. There is also known an image processing apparatus that can scan a newspaper or a book in an unfolded state (see, for example, Patent Document 1). According to the image processing apparatus, articles and photos of necessary pages in newspapers and books can be designated with fingers and converted into data for scrapping. In this case, data such as articles and photographs are stored in the storage device.

  When scrapping articles and photos, it is also common to scrap the source part. This makes it possible to easily recognize the source of scrapped articles and photos at a later date. When scrapping articles and photos as data, it is desirable to also scrap information included in the source part. For example, a technique for imaging a document and extracting bibliographic items of the document from the captured image is known (see, for example, Patent Documents 2 and 3).

JP 2013-250879 A Japanese Patent Laid-Open No. 11-184894 JP-A-8-287189

  By the way, when scraping data such as articles and photographs together with the information of the source part, there is a problem that it is very time-consuming to specify the information and the data each time the page is turned and to scrape together. Patent Documents 2 and 3 described above disclose techniques for extracting bibliographic items, but do not disclose extracting bibliographic items when scraping data such as articles and photographs.

  Therefore, in one aspect, an object of the present invention is to provide an image processing method capable of efficiently associating and storing information included in a specified portion when storing data such as articles and photographs. .

  In the image processing method disclosed in the present specification, when a feature specifying a part of the first imaging target is detected from a captured image obtained by imaging the first imaging target, the detected feature is detected. 1st data which specifies the 1st part in the picked-up image of the said 1st pick-up object specified by and stores the information contained in the specified said 1st part about the picked-up image of the 1st pick-up object. And a second portion corresponding to the first portion in the captured image obtained by capturing the second imaging target having the same contour as the first imaging target is specified. The computer executes a process of storing the information included in the specified second part in the storage unit in association with the second data stored with respect to the captured image of the second imaging target. This is an image processing method.

  In the image processing method disclosed in this specification, an edge detection process is performed on a captured image obtained by imaging an imaging target with an imaging device to detect an outline of the imaging target, and a part of the imaging target is specified. A specified location to be detected is detected based on the position of a specific subject reflected in the captured image of the imaging target, a relative positional relationship between the detected contour and the specified location is calculated, and edge detection processing is performed on another imaging target Information included in a part of the captured image of the other imaging target, which is specified based on the contour of the other imaging target obtained by executing the above and the calculated relative positional relationship. The image processing method is characterized in that a computer executes a process of storing in a storage unit in association with data to be stored regarding a captured image to be captured.

  According to the image processing method disclosed in this specification, when data such as articles and photographs are stored, information included in a designated portion can be efficiently associated and stored.

FIG. 1 is a diagram for explaining an example of an image processing system. FIG. 2 is a diagram for explaining an example of an operation with a finger on a paper medium. FIG. 3 is a diagram for explaining an example of an operation with a finger on a newspaper. FIG. 4 is a diagram for explaining another example of an operation with a finger on a newspaper. FIG. 5 is an example of a block diagram of the image processing apparatus. FIG. 6 is an example of a hardware configuration of the image processing apparatus. FIG. 7 is a flowchart illustrating an example of the operation of the image processing system. FIG. 8 is a diagram for explaining an example of character density. FIG. 9 is a flowchart illustrating another example of the operation of the image processing system. FIG. 10 is a diagram for explaining confirmation or correction of the source of a book. FIG. 11 is a diagram for explaining an example of an operation with a finger on a book. FIG. 12 is a diagram for explaining another example of an operation with a finger on a book. FIG. 13 is another example of a method for determining the type of paper medium.

  Hereinafter, an embodiment for carrying out this case will be described with reference to the drawings.

FIG. 1 is a diagram for explaining an example of the image processing system S.
The image processing system S includes a projection unit 100, a first imaging unit 200, a second imaging unit 300, and an image processing device 400. The image processing apparatus 400 is communicably connected to the projection unit 100, the first imaging unit 200, and the second imaging unit 300 via wired or wireless communication. The image processing apparatus 400 includes a communication unit (not shown), and can use network resources by bidirectionally transmitting / receiving data to / from various external apparatuses via a communication line. Furthermore, the projection part 100 should just be included in the image processing system S as needed, and is not necessarily a required component. As illustrated in FIG. 1, the projection unit 100, the first imaging unit 200, and the second imaging unit 300 are installed in a vertically downward direction, for example, above the projection surface 10. As the projection surface 10, for example, there is a work table for performing various operations such as a counter, a table, and a desk.

  The projection unit 100 projects an image near the paper medium D that is an example of an imaging target. For example, when the paper medium D is placed on the projection surface 10 and a characteristic operation (for example, an operation by a finger FG described later) for designating a part of the paper medium D is performed, the first imaging unit 200 and the second imaging unit 200 are arranged. The imaging unit 300 images the paper medium D together with characteristic operations, and the projection unit 100 projects an image at a predetermined position near the paper medium D based on the captured image. Although details will be described later, for example, when the paper medium D is a book, the projection unit 100 projects an image representing the source of the book. Such a projection method is called Augmented Reality (AR), for example. Further, when the user presents the finger FG on the image, an interaction operation is performed on the projected image, and another image is projected. For example, when the finger FG is presented on an image instructing correction of the content of the source, an image representing the keyboard is projected as another image. The projection unit 100 includes a device that displays an image, such as a projector, and the paper medium D includes, for example, a newspaper NP, a book BK including books, magazines, and books, a booklet, a pamphlet, and the like.

  The first imaging unit 200 and the second imaging unit 300 have the same internal parameters, are known to have parallel optical axes, and are included in a plurality of images captured by the first imaging unit 200 and the second imaging unit 300. This is a so-called parallel stereo arrangement in which arbitrary horizontal axes are arranged on the same straight line. The first imaging unit 200 captures a first image of a region included in the range of the first visual field V1. The second imaging unit 300 captures a second image of a region included in the range of the second visual field V2. Therefore, the first image and the second image each include the projection surface 10 including the paper medium D. In particular, when the finger FG is included on the paper medium D, the image of the finger FG is included in each of the first image and the second image. The first imaging unit 200 and the second imaging unit 300 capture an image including color information of the first image and the second image, and information such as the depth of the paper medium D and the depth of the finger FG. Examples of the first imaging unit 200 and the second imaging unit 300 include imaging devices such as a charge coupled device (CCD) camera and a complementary metal oxide semiconductor (CMOS) camera. Parallel stereo is an arbitrary reference point based on the principle of triangulation using the positional deviation (parallax) of pixels in which the same object is photographed from images captured by two cameras with known and parallel positional relationships. This is a method for calculating the distance from the object to the object.

  The image processing apparatus 400 acquires the first image and the second image captured by the first imaging unit 200 and the second imaging unit 300 with different optical axes, performs various image processing described later, and operates the projection unit 100. To control. Thereby, the projection unit 100 projects various images. For example, the position of the user's finger FG in the world coordinate system (on three dimensions) is calculated based on the first image and the second image including the acquired characteristic motion. When the depth at the calculated position is included within a predetermined distance (for example, 0 to 10 mm) from the paper medium D, an image is projected near the paper medium D. If the calculated position is included in the projection area of the image, another image is projected. Here, the arrangement position of the image processing apparatus 400 in the image processing system S can be freely set. For example, the image processing apparatus 400 may be disposed at the same position as the projection unit 100. Details of the functions of the image processing apparatus 400 will be described later.

  By the way, at the time of the configuration of the image processing system S shown in FIG. 1, alignment (calibration) in the coordinate system of the first imaging unit 200 and the second imaging unit 300 and the world coordinate system of the projection unit 100 is performed in advance. It shall be. In addition, when the positional relationship among the first imaging unit 200, the second imaging unit 300, and the projection unit 100 is changed after the use of the image processing apparatus 400 is started, the calibration may be executed at least once. Here, as an example of a specific calibration method, an arbitrary projection image projected by the projection unit 100 is captured by the first image capturing unit 200 and the second image capturing unit 300, thereby calibrating the image processing apparatus 400. Explain how to execute the application. In this method, it is assumed that calibration is performed in advance for each of the first imaging unit 200 and the second imaging unit 300.

First, the projecting unit 100 projects an arbitrary marker on an arbitrary coordinate value (x _p , y _p ) in the world coordinate system. The marker can use any color or shape that can be easily distinguished from the surrounding background. Then, the first image capturing unit 200 and the second image capturing unit 300 image the marker projected on the predetermined projection plane 10. Subsequently, the image processing apparatus 400 recognizes the marker by any known image processing. For example, when the projection unit 100 projects a circular pattern as a marker, the image processing apparatus 400, for example, “Kimme et al.,“ Finding circles by an array of accumulators ”, Communications of the Association for Computing Machinery, # 18, pp .120-122, 1975. ”, the circular shape can be recognized by the Hough circle transformation. Here, the coordinate value when the image processing apparatus 400 recognizes the marker is assumed to be (x _i , y _i ). The image processing apparatus 400 repeats the above processing for four points at an arbitrary place. The image processing apparatus 400 converts each component of the 3 × 3 homography matrix H from four sets of (x _i , y _i ) corresponding to (x _p , y _p ) obtained by the processing, Calculation is performed using an 8-dimensional simultaneous linear equation. Note that homography is a matrix representing projective transformation from one plane in a three-dimensional space to another plane. In the present embodiment, the image processing apparatus 400 obtains an association between the coordinate planes of the first imaging unit 200 and the second imaging unit 300 and the projection unit coordinate plane of the projection unit 100. The image processing apparatus 400 can use the homography matrix at the time of image projection by storing the calculated homography matrix in, for example, a cache or a memory (not shown).

  FIG. 2 is a diagram for explaining an example of an operation with the finger FG on the paper medium D.

  As shown in FIG. 2A, the user touches the paper medium D with the finger FG (the time of touch is defined as time t1), and the finger FG is slid to the position of time t2, or the finger FG is moved to the time t2. The finger FG is then detached from the paper medium D. A rectangular area 50 having a line segment based on the position of the finger FG at time t2 as a diagonal line from the position of the finger FG at time t1 is specified as a designated location, and image information of the rectangular area 50 as shown in FIG. Are stored in a storage unit 410 (described later) included in the image processing apparatus 400 as a scanned image. Optical character recognition (OCR: optical character recognition) may be performed on the image information of the rectangular area 50 and stored in the storage unit 410 as character information. Note that the contact and separation of the finger FG with respect to the paper medium D can be determined based on, for example, the calculated depth of the finger FG. For example, the position of the finger FG in the coordinate system of the first imaging unit 200 and the second imaging unit 300 is shown in different positions on the paper medium D in the first image and the second image, and is defined from the positions. The depth of the fingertip is calculated by the binocular stereo method based on the parallax. Based on the calculated depth of the finger FG, it is possible to determine whether the finger FG is in contact with or separated from the finger FG.

  FIG. 3 is a diagram for explaining an example of the operation with the finger FG for the newspaper NP. FIG. 4 is a diagram for explaining another example of the operation with the finger FG for the newspaper NP.

  As shown in FIG. 3A, when the first page of the newspaper NP is placed on the projection surface 10, the first imaging unit 200 and the second imaging unit 300 capture the first page of the newspaper NP. The first page of the newspaper NP is an example of a first imaging target. When the user's finger FG touches the newspaper NP and makes a characteristic movement that designates the source portion 60 that is a part of the first page of the newspaper NP, the first image by the first imaging unit 200 and the second image by the second imaging unit 300 are displayed. The characteristic movement is detected from the two images, and the designated source portion 60 is specified. Then, the image information of the specified source portion 60 or the character information “August 18, 2014 XX newspaper” included in the source portion 60 is held in the image processing apparatus 400.

  Next, when the user's finger FG touches the newspaper NP and makes a characteristic movement that designates a scrap target portion 70 that is different from the source portion 60 on the first page of the newspaper NP, the first imaging unit 200 The characteristic movement is detected from the first image and the second image by the second imaging unit 300, and the designated scrap target portion 70 is specified. Then, the specified image data of the scrap target portion 70 is stored in the storage unit 410 as shown in FIG. 3B in association with image information or character information held in the image processing apparatus 400. . Thereby, the scrap for the first page of the newspaper NP is completed.

  Next, when the user desires scrap for another page of the newspaper NP, the user turns the page of the newspaper NP and places the eighth page of the newspaper NP on the projection plane 10, for example, as shown in FIG. The eighth page of the newspaper NP is an example of a second imaging target. When the eighth page having the same outline as the first page is placed on the projection surface 10, the first imaging unit 200 and the second imaging unit 300 image the eighth page of the newspaper NP. At this time, on the eighth page of the newspaper NP, even if the source portion 80 corresponding to the source portion 60 of the first page of the newspaper NP is not designated by the finger FG by the image processing apparatus 400, the source portion 80 is designated by the finger FG. It is dynamically specified based on the relative positional relationship between the position of the source portion 60 and the outline of the first page of the newspaper NP. Note that whether or not the contours are the same is determined by an edge detection process performed by a later-described first part specifying unit 430 and second part specifying unit 440 included in the image processing apparatus 400. The edge detection process is a process for specifying a portion where the brightness of the image changes sharply or discontinuously. For example, when the paper medium D is placed on the projection surface 10, the boundary between the projection surface 10 and the paper medium D changes sharply. In the edge detection process, the contour of the paper medium D is detected by specifying a sharply changing portion.

  Therefore, when the user's finger FG touches the newspaper NP and makes a characteristic movement that specifies the scrap target portion 90 of the eighth page of the newspaper NP, the first image by the first imaging unit 200 and the second imaging unit 300 The characteristic motion is detected from the second image, and the designated scrap target portion 90 is specified. Then, the image data of the specified scrap target portion 90 is associated with the image information or character information “August 18, 2014 XX newspaper” of the dynamically specified source portion 80, and FIG. ) As shown in FIG. Thereby, the scrap for the eighth page of the newspaper NP is completed.

  As described above, according to the image processing apparatus 400 according to the present embodiment, when data related to a captured image including an article or a photograph is stored, the information included in the designated portion is efficiently associated and stored. Can do.

  Next, the details of the image processing apparatus 400 will be described with reference to FIG.

FIG. 5 is an example of a block diagram of the image processing apparatus 400.
As illustrated in FIG. 5, the image processing apparatus 400 includes a storage unit 410, a feature detection unit 420, a first partial identification unit 430 that is an example of a first identification unit, and a second that is an example of a second identification unit. The part specifying unit 440 and the storage unit 450 as an example of the storage unit are included.

  The storage unit 410 stores the image information or character information of the source portions 60 and 80 described with reference to FIGS. 3 and 4 and the scrap target portions 70 and 90 in association with each other.

  The feature detection unit 420 detects a feature specifying a part of the paper medium D from the first image and the second image captured by the first imaging unit 200 and the second imaging unit 300, respectively. As described above, the feature of designating a part of the paper medium D is an operation of sliding or tracing the finger FG while the user's finger FG touches the paper medium D, and then releasing the finger FG from the paper medium D. . Instead of the finger FG, a rod-shaped member including a writing instrument may be used as a specific subject. When the feature detection unit 420 detects the feature of the finger FG, the feature detection unit 420 transmits the first image and the second image to the first part specifying unit 430 and the second part specifying unit 440.

  When the first part specifying unit 430 receives the first image and the second image transmitted from the feature detection unit 420, the first part specifying unit 430 specifies a part designated by the finger FG in the first image and the second image. For example, when the paper medium D is newspaper NP, the first part specifying unit 430 specifies the rectangular source part 60 and the scrap target part 70 specified by the finger FG. When the source part 60 is specified, the first part specifying unit 430 holds image information representing the source part 60 or character information included in the source part 60. When the first part specifying unit 430 specifies the scrap target part 70, the first part specifying unit 430 associates the stored image information or character information with the image data of the scrap target part 70 and transmits the image information or character information to the storage unit 450.

  The second part specifying unit 440 holds the first image and the second image transmitted from the feature detection unit 420. When another first image and another second image are transmitted from the feature detection unit 420, the second part specifying unit 440 is different from the held first image, the different first image, and the held second image. Are compared with each other, the page of the paper medium D specified by the held first image and the second image, and the page of the paper medium D specified by another first image and another second image. Are the same. When determining that the page of the paper medium D is different, the second part specifying unit 440 determines that the page of the paper medium D is turned. When it is determined that the page of the paper medium D has been turned, the second part specifying unit 440 specifies a part designated by the finger FG in another first image and another second image. For example, when the paper medium D is a newspaper NP, the second part specifying unit 440 dynamically specifies the source part 80 corresponding to the source part 60 and also selects the rectangular scrap target part 90 specified by the finger FG. Identify. The second part specifying unit 440 transmits the image information or character information of the specified source part 80 and the image data of the scrap target part 90 to the storage unit 450.

  The storage unit 450 stores the image information or character information of the source part 60 and the image data of the scrap target part 70 transmitted from the first part specifying unit 430 in the storage unit 410. As a result, the scrap target portion 70 associated with the source portion 60 is stored in the storage unit 410. The storage unit 450 stores the source part 80 and the scrap target part 90 transmitted from the second part specifying unit 440 in the storage unit 410. As a result, the scrap target portion 90 associated with the source portion 80 is stored in the storage unit 410.

  Next, the hardware configuration of the image processing apparatus 400 will be described with reference to FIG.

FIG. 6 is an example of a hardware configuration of the image processing apparatus 400.
As shown in FIG. 6, the image processing apparatus 400 includes at least a central processing unit (CPU) 400A, a random access memory (RAM) 400B, a read only memory (ROM) 400C, and a network I / F (interface) 400D. . The image processing apparatus 400 may include at least one of a hard disk drive (HDD) 400E, an input I / F 400F, an output I / F 400G, an input / output I / F 400H, and a drive apparatus 400I as necessary. The CPU 400A... Drive device 400I is connected to each other by an internal bus 400J. At least the CPU 400A and the RAM 400B cooperate to realize a computer.

An input device 510 is connected to the input I / F 400F. Examples of the input device 510 include a keyboard and a mouse.
A display device 520 is connected to the output I / F 400G. An example of the display device 520 is a liquid crystal display.
A semiconductor memory 530 is connected to the input / output I / F 400H. Examples of the semiconductor memory 530 include a universal serial bus (USB) memory and a flash memory. The input / output I / F 400H reads a program and data stored in the semiconductor memory 530.
The input I / F 400F and the input / output I / F 400H include, for example, a USB port. The output I / F 400G includes a display port, for example.

A portable recording medium 540 is inserted into the drive device 400I. Examples of the portable recording medium 540 include a removable disk such as a Compact Disc (CD) -ROM and a Digital Versatile Disc (DVD). The drive device 400I reads a program and data recorded on the portable recording medium 540.
The network I / F 400D includes, for example, a port and a physical layer chip (PHY chip). The image processing apparatus 400 is connected to the network via the network I / F 400D.

  In the RAM 400B described above, the program stored in the ROM 400C or the HDD 400E is stored by the CPU 400A. In the RAM 400B, the program recorded on the portable recording medium 540 is stored by the CPU 400A. When the stored program is executed by the CPU 400A, various operations described later are executed. In addition, what is necessary is just to make a program according to the flowchart mentioned later.

  Next, the operation of the above-described image processing system S will be described with reference to FIGS.

  FIG. 7 is a flowchart illustrating an example of the operation of the image processing system S. More specifically, FIG. 7 is a flowchart including an operation in which the image processing apparatus 400 specifies the source portion 60. FIG. 8 is a diagram for explaining an example of character density. FIG. 9 is a flowchart illustrating another example of the operation of the image processing system S. More specifically, FIG. 9 is a flowchart showing an example of an operation in which the image processing apparatus 400 identifies the scrap target portion 70 and associates it with the source portion 60 and associates the scrap target portion 90 with the source portion 80.

  First, as shown in FIG. 7, the first imaging unit 200 and the second imaging unit 300 image the paper medium D on the projection surface 10 (step S101). Next, when the process of step S101 is completed, the feature detection unit 420 calculates the character density of the paper medium D based on the first image by the first imaging unit 200 and the second image by the second imaging unit 300 (step S101). S102), it is determined whether or not the calculated character density is 20% or more (step S103). The character density is a ratio of characters to the paper surface of the paper medium D. As shown in FIG. 8A, when the paper medium D is a newspaper NP, the character density is higher than that of the cover of the book BK shown in FIG. 8B. Therefore, when the calculated character density is 20% or more (step S103: YES), the feature detection unit 420 determines that the paper medium D is a newspaper NP. If the calculated character density is less than 20% (step S103: NO), the feature detection unit 420 determines that the paper medium D is a book BK. The threshold for determining the character density is not limited to 20%, and may be changed as appropriate.

  When it is determined that the paper medium D is the newspaper NP, the first part specifying unit 430 then specifies the source part 60 specified by the finger FG (step S104) and is included in the specified source part 60. Character information is extracted (step S105). Thereby, the character information “August 18, 2014 XX newspaper” (see FIG. 3A) is extracted. In step S104 or step S105, the first part specifying unit 430 detects the outline of the first page of the newspaper NP and also detects the location of the source part 60 designated by the position of the finger FG. The relative positional relationship between the detected contour and the location of the source portion 60 is calculated and held. When the process of step S105 is completed, the first part specifying unit 430 then holds the character information as an index (step S106) and ends the process of specifying the source part 60.

  Next, as shown in FIG. 3, when the rectangular scrap target portion 70 is designated by the finger FG, the movement of the finger FG is detected by the feature detection unit 420, and as shown in FIG. 9, the first part identification is performed. The unit 430 reads image data related to the rectangular scrap target portion 70 designated by the finger FG (step S201). When reading the image data, the first part specifying unit 430 associates the index held by the first part specifying unit 430 with the image data (step S202). Here, the first part specifying unit 430 determines whether or not the paper medium D is newspaper NP (step S203). In the present embodiment, since the paper medium D is determined to be newspaper NP by the process of step S103, the process of the subsequent step S204 is skipped. Therefore, when the process of step S203 is completed, the storage unit 450 stores the index and the image data in the storage unit 410 (step S205). Thereby, as shown in FIG.3 (b), the image data regarding the scrap object part 70 is preserve | saved in the memory | storage part 410 with an index.

  Next, the second part specifying unit 440 determines whether or not the paper medium D has been changed to another paper surface (step S206). This determination is made based on, for example, a difference in captured images or a difference in page numbers included in the captured images. When it is determined that the paper medium D has been changed to another paper surface (step S206: YES), the second part specifying unit 440 determines again whether the paper medium D is newspaper NP (step S206). S207). As described above, in the present embodiment, it is determined that the paper medium D is the newspaper NP by the process of step S103. Accordingly, the second part specifying unit 440 determines that the paper medium D is the newspaper NP (step S207: YES), and specifies the source part 80 corresponding to the index (step S208). When specifying the source part 80, the second part specifying unit 440 uses the outline of the eighth page of the newspaper NP and the relative positional relationship held.

  Further, the second part specifying unit 440 reads image data related to the rectangular scrap target part 90 designated by the finger FG (step S209). When the second part specifying unit 440 reads the image data, the source part 80 held by itself is associated with the image data (step S210), and the storage unit 450 stores the character information and the image data of the source part 80. 410 is stored (step S211). Thereby, as shown in FIG. 4B, the image data related to the scrap target portion 90 is stored in the storage unit 410 together with the character information of the source portion 80. When the processing of step S211 is completed, the second part specifying unit 440 determines again whether the paper surface of the paper medium D has been changed, and determines that the paper surface of the paper medium D has not been changed ( Step S206: NO), the process ends.

  In this way, when the newspaper NP is used as the paper medium D and scrap is performed, even if the source portion is not specified every time the page of the newspaper NP is turned, the source portion is specified first. In the subsequent scraps, by specifying the scrap target part, the part related to the source is dynamically associated. Therefore, it is possible to greatly reduce the trouble of associating the source part with scraps of articles, photographs, illustrations and the like.

  Next, a case where the book BK is used as the paper medium D will be described with reference to FIGS.

  In the process of step S103 in FIG. 7 described above, when the calculated character density is less than 20% and it is determined that the paper medium D is the book BK, the first part specifying unit 430 determines whether the book BK is in the book BK. The maximum character is extracted and a title is temporarily set (step S107). As a result, among the characters attached to the cover of the book BK shown in FIG. 8B, “animal ecology”, which is the maximum character, is extracted and temporarily set as the title.

  Next, the first part specifying unit 430 extracts the minimum character at a position other than the lowest level in the book BK, and temporarily sets the author (step S108). As a result, “ΔΔΔΔ”, which is the smallest character at a position other than the lowest level, is extracted from the characters attached to the cover of the book BK shown in FIG. 8B, and is temporarily set to the author. The first part specifying unit 430 removes and extracts the largest character and “author” attached next to “ΔΔΔΔ” when extracting. Next, the first part specifying unit 430 extracts the character at the lowest position in the book BK, and temporarily sets the publisher (step S109). As a result, “xx publication”, which is the character at the lowest position, is extracted from the characters attached to the cover of the book BK shown in FIG. 8B and is temporarily set to the publisher. When the first part specifying unit 430 finishes the process of step S109, the first part specifying unit 430 transmits a message to that effect to the projection unit 100, and the projection unit 100 projects the temporarily set content (step S110). As a result, as shown in FIG. 10 (a), the provisionally set title image “Animal Ecology”, author “△△△△” and publisher “XX Publishing” are included in the source image 40 and the source image 40. The confirmed image 41 for confirming the content to be read and the corrected image 42 for correcting the content are projected at a predetermined position near the book BK.

  The first part specifying unit 430 determines whether or not the content included in the source image 40 is modified (step S111). For example, as illustrated in FIG. 10B, when the finger FG is presented to the corrected image 42 and the corrected image 42 on the projection surface 10 is touched, the first part specifying unit 430 is performed by the first imaging unit 200. Based on the first image and the depth of the finger FG included in the second image by the second imaging unit 300, it is determined that the content included in the source image 40 is corrected (step S111: YES). In this case, the projection unit 100 projects the keyboard image 43 as shown in FIG. 10C (step S112). The user corrects the title, author, and publisher included in the source image 40 by presenting the finger FG to the keyboard image 43 and appropriately touching each key of the keyboard image 43 on the projection surface 10. If the accuracy of the OCR process is low, an accurate title or the like may not be stored. However, since the title can be corrected, an accurate source can be stored in the storage unit 410.

  The first part specifying unit 430 modifies the content included in the source image 40 based on the depth of the finger FG included in the first image by the first imaging unit 200 and the second image by the second imaging unit 300. Project. For example, when the finger FG is presented to the confirmed image 41 and the confirmed image 41 on the projection plane 10 is touched, the first part specifying unit 430 is generated by the first image by the first imaging unit 200 and the second imaging unit 300. Based on the depth of the finger FG included in the second image, it is determined that the content included in the source image 40 is not corrected (step S111: NO). When it is determined that there is no correction in the process of step S111, the first part specifying unit 430 executes the process of step S106. Thereby, the content included in the source image 40 is held as an index. As described above, when the book BK is used as the paper medium D, the content is dynamically extracted even if the content included in the source image 40 is not specified by the finger FG.

  Next, as shown in FIG. 11A, when the rectangular scrap target portion 20 is designated by the finger FG, the movement of the finger FG is detected by the feature detection unit 420, and the first part specifying unit 430 As shown in FIG. 9, image data relating to the rectangular scrap target portion 20 designated by the finger FG is read (step S201). When reading the image data, the first part specifying unit 430 associates the index held by the first part specifying unit 430 with the image data (step S202). Here, the first part specifying unit 430 determines whether or not the paper medium D is newspaper NP (step S203). In the present embodiment, since the paper medium D is determined to be the book BK by the process of step S103, the first part specifying unit 430 reads the numerical information below the paper medium D and associates it with the saved index. (Step S204). It should be noted that the numerical information part indicating the page number written at the bottom of the paper medium D may be designated with the finger FG. Thereby, even when moving to another page, the part corresponding to the part is specified later, and the labor of associating numerical information is saved. When the process of step S204 is completed, the storage unit 450 stores the index and image data associated with the numerical information in the storage unit 410 (step S205). As a result, as shown in FIG. 11B, the image data regarding the scrap target portion 20 is stored in the storage unit 410 together with the index and the page number.

  Next, the second part specifying unit 440 determines whether or not the paper surface of the paper medium D has been changed (step S206). As shown in FIG. 12, when the page of the book BK is changed from the 10th page as the first imaging target to the 15th page as the second imaging target, the second part specifying unit 440 It is determined that the paper surface of the medium D has been changed (step S206: YES). In this case, it is determined again whether or not the paper medium D is the newspaper NP (step S207). As described above, in the present embodiment, it is determined that the paper medium D is the book BK by the process of step S103. Accordingly, the second part specifying unit 440 determines that the paper medium D is the book BK (step S207: NO), and repeats the processing from steps S201 to S205.

  As a result, as shown in FIG. 12A, when the rectangular scrap target portion 21 is designated by the finger FG, the motion of the finger FG is detected by the feature detection unit 420, and the rectangular shape designated by the finger FG is detected. Image data relating to the scrap target portion 21 is read. When the image data is read, an index held by itself, that is, a title, an author, and a publisher included in the source image 40 are associated with the image data. Further, in the present embodiment, it is determined that the paper medium D is the book BK. Therefore, the numerical information at the bottom of the paper medium D is read and associated with the held index, and the index, numerical information, and image data are stored in the storage unit 410. Thereby, as shown in FIG. 12B, the image data regarding the scrap target portion 21 is stored in the storage unit 410 together with the index and the page number.

  As described above, even when the book BK is used as the paper medium D, information about the source included in the dynamically extracted portion is efficiently stored when storing the image data regarding the captured image including articles and photographs. It can be stored in association.

  In the embodiment described above, the character density is used when determining whether the paper medium D is the newspaper NP or the book BK. For example, as shown in FIG. If it is registered in the image processing apparatus 400 and the size of the paper medium D is equal to or larger than the size of the frame 30, it is determined as newspaper NP. Conversely, if the size of the paper medium D is less than the size of the frame 30, it is determined as a book BK. It is possible to determine whether the paper medium D is the newspaper NP or the book BK also by such a method.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to the specific embodiments according to the present invention, and various modifications are possible within the scope of the gist of the present invention described in the claims.・ Change is possible. For example, the touch wording described above may include a touch including a hovering function that can be moved by holding the finger FG over the image without directly touching the image with the finger FG. Further, the scrap target portions 70 and 90 in the newspaper NP and the scrap target portions 20 and 21 in the book BK may be the entire paper surface instead of a part of the paper surface.

In addition, the following additional notes are disclosed regarding the above description.
(Appendix 1) When a feature that specifies a part of the first imaging target is detected from a captured image obtained by imaging the first imaging target, the first feature that is specified by the detected feature is detected. A first portion in the captured image of the imaging target is identified, and information included in the identified first portion is associated with first data to be stored regarding the captured image of the first imaging target in the storage unit. The second portion corresponding to the first portion is specified in the captured image obtained by capturing and capturing the second imaging target having the same outline as the first imaging target, and the identified second An image processing method, wherein a computer executes a process of storing information included in a part in the storage unit in association with second data stored with respect to a captured image of the second imaging target.
(Supplementary Note 2) The first imaging target and the second imaging target are paper media, and the first part and the second part include information on the source or page number of the paper medium. The first data is data of a third part different from the first part in the first imaging target, and the second data is the second data in the second imaging target. The image processing method according to appendix 1, wherein the data is a data of a fourth portion different from the portion of 2.
(Additional remark 3) Based on the size and position of the character which appears in the captured image obtained by imaging the 3rd imaging target which is a cover of a paper medium, the source of the paper medium is specified, and the specified source is The image processing method according to appendix 1 or 2, wherein the first data and the second data are stored in the storage unit in association with each other.
(Supplementary Note 4) Edge detection processing is performed on a captured image obtained by imaging the imaging target with the imaging device to detect the contour of the imaging target, and a designated portion for designating a part of the imaging target is captured. Detected by the position of a specific subject in the captured image of the target, obtained by calculating the relative positional relationship between the detected contour and the specified location, and performing edge detection processing on other imaging targets Information included in a part of the captured image of the other imaging target, which is specified based on the contour of the other imaging target and the calculated relative positional relationship, is stored for the captured image of the other imaging target. An image processing method, wherein a computer executes a process of storing in a storage unit in association with data to be performed.
(Supplementary Note 5) When a feature that specifies a part of the first imaging target is detected from a captured image obtained by imaging the first imaging target, the first feature specified by the detected feature is detected. A first portion in the captured image of the imaging target is identified, and information included in the identified first portion is associated with first data to be stored regarding the captured image of the first imaging target in the storage unit. The second portion corresponding to the first portion is specified in the captured image obtained by capturing and capturing the second imaging target having the same outline as the first imaging target, and the identified second An image processing program for causing a computer to execute a process of storing information included in a portion in association with second data stored with respect to a captured image of the second imaging target and storing the information in the storage unit.
(Supplementary Note 6) When a feature that specifies a part of the first imaging target is detected from a captured image obtained by imaging the first imaging target, the first feature specified by the detected feature is detected. First identification means for identifying a first portion in a captured image of the imaging target, and the first in the captured image obtained by capturing a second imaging target having the same contour as the first imaging target. A second specifying unit that specifies a second part corresponding to the part, and information included in the specified first part is associated with first data to be stored for the captured image of the first imaging target. Storage means for storing in the storage unit in association with the second data to be stored in the storage unit and the information included in the specified second part stored with respect to the captured image of the second imaging target An image processing apparatus.
(Supplementary note 7) The first imaging object and the second imaging object are paper media, and the first part and the second part include information on the source or page number of the paper medium. The first data is data of a third part different from the first part in the first imaging target, and the second data is the second data in the second imaging target. The image processing apparatus according to appendix 6, wherein the image processing apparatus is data of a fourth part different from the part of 2.
(Supplementary Note 8) The first specifying means specifies the source of the paper medium based on the size and position of the character appearing in the captured image obtained by imaging the third imaging target that is the cover of the paper medium. The image processing apparatus according to appendix 6 or 7, wherein the identified source is stored in the storage unit in association with the first data and the second data, respectively.

DESCRIPTION OF SYMBOLS S Image processing system 100 Projection part 200 1st imaging part 300 2nd imaging part 400 Image processing apparatus 410 Storage part 420 Feature detection part 430 1st partial specific part 440 2nd partial specific part 450 Storage part

Claims (4)

When a feature specifying a part of the first imaging target is detected from a captured image obtained by imaging the first imaging target, imaging of the first imaging target specified by the detected feature is performed. Identify the first part in the image,
The information included in the identified first part is stored in the storage unit in association with the first data to be stored regarding the captured image of the first imaging target,
Identifying a second portion corresponding to the first portion in a captured image obtained by capturing a second imaging target having the same outline as the first imaging target;
Storing the information included in the identified second part in the storage unit in association with the second data stored with respect to the captured image of the second imaging target;
An image processing method, wherein the computer executes the processing. The first imaging target and the second imaging target are paper media,
The first part and the second part include information on the source or page number of the paper medium,
The first data is data of a third part different from the first part in the first imaging target,
The second data is data of a fourth part different from the second part in the second imaging target.
The image processing method according to claim 1. Identifying the source of the paper medium based on the size and position of the characters appearing in the captured image obtained by imaging the third imaging target that is the cover of the paper medium;
Storing the identified source in the storage unit in association with the first data and the second data, respectively.
The image processing method according to claim 1, wherein: An edge detection process is performed on a captured image obtained by imaging the imaging target with an imaging device to detect the contour of the imaging target;
A designated portion for designating a part of the imaging target is detected by a position of a specific subject reflected in the captured image of the imaging target;
Calculating the relative positional relationship between the detected contour and the specified location;
A part of the captured image of the other imaging target specified based on the contour of the other imaging target obtained by executing the edge detection process on the other imaging target and the calculated relative positional relationship Storing the information included in the storage unit in association with the data to be stored regarding the captured image of the other imaging target,
An image processing method, wherein the computer executes the processing.

Images