JP2012055412A - Image processor, endoscope apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor enhancing the allowable range of the relative movement between the distal end of a scope and an object to be observed in magnification observation, an endoscope apparatus, an image processing method, and a program.SOLUTION: The image processing apparatus includes the image forming and processing section 310 obtaining a plurality of images obtained by imaging the object to be observed by magnifying the same at a plurality of different magnifications, the targeted magnified image selecting section 325 selecting the first to the nth magnified images with relatively different magnifications out of the plurality of image is, and the position determining section 330 among different magnification images selecting a reference image and an image to be detected with magnification higher than the reference image out of the first to the nth images, detecting the range on the reference image corresponding to the image to be detected, and correlating the image to be detected to the range.

Description

  The present invention relates to an image processing apparatus, an endoscope apparatus, an image processing method, a program, and the like.

  In endoscopic diagnosis, there is a demand for improving the accuracy of detecting a lesion in a body cavity. As a general technique for improving accuracy, an endoscope (magnifying endoscope) equipped with a magnifying optical system is used to magnify and observe the difference in tissue between the lesioned part and the normal part at a magnification equivalent to that of a microscope. It has been known. Such a magnifying endoscope has an enlargement ratio of, for example, several tens to several hundred times, and is used for observation of a pit pattern or a blood vessel running pattern. It is known that there is a difference in these patterns between the lesioned part and the normal part, which is one criterion for lesion diagnosis.

JP 2001-46331 A

  However, in such magnified observation, the field of view becomes very narrow due to the large magnification. For this reason, there is a problem that it is difficult to determine which position of the observation target is observed even if a relatively small movement occurs between the scope and the observation target. In this way, when the observation position is lost, it is necessary to repeat the procedure of stopping the enlarged observation, returning to the normal observation with a small enlargement ratio, aligning the region to be observed, and performing the enlarged observation again. Therefore, the diagnosis time is extended and the patient is burdened.

  For example, Patent Document 1 calculates a correction curve angle for offsetting image shift caused by relative movement between the scope tip and the observation target, and curves the scope tip at the correction curve angle. A method for correcting image shift is disclosed. In this method, when the relative movement between the scope tip and the observation target is small, it is possible to correct the image shift. However, with this method, when an image shift that exceeds the field of view of the captured image occurs, the image shift cannot be calculated from the captured image, and thus there is a problem that the correction curve angle cannot be calculated. Further, in this method, since the allowable deviation amount becomes smaller as the magnification becomes higher, the magnifying endoscope with higher magnification becomes less practical.

  According to some aspects of the present invention, there are provided an image processing device, an endoscope device, an image processing method, a program, and the like that improve an allowable range of relative movement between a distal end of a scope and an observation target in magnified observation. it can.

  One aspect of the present invention is an image acquisition unit that acquires a plurality of images obtained by enlarging and imaging an observation object at a plurality of different magnifications, and a magnification between the images from the plurality of images. An image selection unit that selects first to nth magnification images (n is a natural number of n ≧ 2) that are relatively different from each other, and a reference image and a higher image than the reference image among the first to nth magnification images. The present invention relates to an image processing apparatus that includes a region detection unit that selects a detection target image with a magnification, detects a region on the reference image corresponding to the detection target image, and associates the detection target image with the region.

  According to an aspect of the present invention, the first to nth magnification images are selected from a plurality of images, and the reference image and the detection target image having a higher magnification than the reference image are selected from the first to nth magnification images. A region on the reference image corresponding to the detection target image is selected, and the detection target image is associated with the region. This makes it possible to improve the allowable range of relative movement between the scope tip and the observation target in magnified observation.

  Another aspect of the present invention relates to an endoscope apparatus including the image processing apparatus described above.

  According to another aspect of the present invention, a plurality of images obtained by enlarging and imaging an observation object at a plurality of different magnifications are acquired, and each of the plurality of images is relatively compared with each other. First to nth magnification images (n is a natural number of n ≧ 2) having different magnifications are selected, and a reference image and a detection target image having a higher magnification than the reference image are selected from the first to nth magnification images. The present invention relates to an image processing method for selecting, detecting an area on the reference image corresponding to the detection target image, and associating the detection target image with the area.

  According to another aspect of the present invention, an image acquisition unit that acquires a plurality of images obtained by enlarging and imaging an observation object at a plurality of different magnifications, and each image from the plurality of images. An image selection unit that selects first to nth magnification images (n is a natural number of n ≧ 3) having relatively different magnifications, and a reference image and the reference image from among the first to nth magnification images This relates to a program that selects a higher magnification detection target image, detects a region on the reference image corresponding to the detection target image, and causes the computer to function as a region detection unit that associates the detection target image with the region. To do.

  According to another aspect of the present invention, an image acquisition unit that acquires a series of moving image images obtained by imaging an observation target and a control for displaying the series of moving image images in a first display area are performed. A display control unit, wherein the display control unit displays a moving image selected from the series of moving images, the image having a lower magnification than the moving image displayed in the first display area. The display control unit performs control to display the image on the second display area as a low-magnification image, and the display control unit is information representing an area on the low-magnification image corresponding to the moving image displayed on the first display area. In addition, the present invention relates to an endoscope apparatus that performs control to display identification information that moves on the low-magnification image in accordance with movement of the visual field range of the series of moving image images in the second display area.

Explanatory drawing about the comparative example of this embodiment. Explanatory drawing about the method of this embodiment. Explanatory drawing about the method of this embodiment. The structural example of an endoscope apparatus. The detailed structural example of a rotation color filter. An example of spectral characteristics of a color filter. 3 is a detailed configuration example of an image processing unit. An example of a display image. 3 is a detailed configuration example of a target magnification image group selection unit. Explanatory drawing about operation | movement of a target magnification image group selection part. Explanatory drawing about operation | movement of a target magnification image group selection part. Explanatory drawing about operation | movement of a target magnification image group selection part. 3 is a detailed configuration example of a position determining unit between different magnification images. Explanatory drawing about operation | movement of the position determination part between different magnification images. Explanatory drawing about operation | movement of the position determination part between different magnification images. The 2nd structural example of an endoscope apparatus. The detailed structural example of a rotation color filter. An example of spectral characteristics of a color filter that transmits white light. An example of spectral characteristics of a color filter that transmits special light. The 2nd detailed structural example of an image process part. A typical configuration example of the scope tip. FIG. 6 is a schematic explanatory diagram regarding imaging timing. 3 is a detailed configuration example of an AF control unit. Explanatory drawing about AF area. The detailed structural example of a tip bending drive part. The system block diagram which shows the structure of a computer system. The block diagram which shows the structure of the main-body part in a computer system. The flowchart example of the process which this embodiment performs. 7 is a detailed flowchart example of target magnification image selection processing. 7 is a detailed flowchart example of area detection processing.

  Hereinafter, this embodiment will be described. In addition, this embodiment demonstrated below does not unduly limit the content of this invention described in the claim. In addition, all the configurations described in the present embodiment are not necessarily essential configuration requirements of the present invention.

1. First, a problem in the above-described enlarged photographing will be described with reference to FIG. As shown at A1 in FIG. 1, at the time of enlargement photographing, the scope scope FA1 becomes very narrow because the scope of the endoscope is photographed close to the observation target Tg. Therefore, as shown in A2, when the motion of the digestive tract or the blurring of the scope occurs, the visual field is greatly shifted even if the motion is small, and the visual field FA2 is easily detached from FA1. In this case, it is difficult to introduce the observation target Tg into the field of view again from only the images that do not overlap FA1 and FA2.

  Therefore, in the present embodiment, an image at the time of low-magnification shooting is acquired, and the positional relationship between the image and the image at the time of high-magnification shooting is detected, whereby the observation target Tg can be reintroduced. The method of this embodiment will be specifically described with reference to FIGS.

  As shown in B1 of FIG. 2, in the enlarged photographing, the scope is generally gradually approached from a position away from the observation target Tg. Then, as shown in B2, the scope is brought closer to the observation target Tg up to the position where the maximum magnification is obtained. The field of view narrows from FB1 to FB2. As shown in B3, even if the visual field FB3 deviates from the observation target Tg, the position of the FB3 can be known within the FB1.

  Specifically, as shown in FIG. 3, matching processing is performed between an image PB1 obtained by imaging the field of view FB1 and images PB2 and PB3 obtained by imaging the fields of view FB2 and FB3 at the maximum magnification. Then, using the result of the matching process, the current shooting range is displayed on the image PB1, or the scope tip is automatically bent in the direction of the observation target Tg. As will be described later, in this embodiment, the matching process between the low-magnification image and the maximum-magnification image is performed in stages using the intermediate magnification image, and high-precision matching is possible even at a high magnification such as 100 times. To.

2. Endoscopic Device FIG. 4 shows a configuration example of the endoscope device of the present embodiment that facilitates reintroduction of the observation target Tg. The endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300 (control device), a display unit 400, and an external I / F unit 500.

  The light source unit 100 has spectral characteristics from the white light source 101, a rotating color filter 102 having a plurality of filters each having a different spectral transmittance, a rotation driving unit 103 that drives the rotating color filter 102, and the rotating color filter 102. A condensing lens 104 that condenses the collected light on the incident end face of the light guide fiber 201.

  As shown in FIG. 5, the rotation color filter 102 includes, for example, three primary color red color filters 601, three primary color blue color filters 602, three primary color green color filters 603, and a rotation motor 803. . FIG. 6 shows examples of spectral characteristics of these three color filters. As shown in FIG. 6, it transmits wavelengths 580 to 700 nm, 400 to 500 nm, and 480 to 600 nm, respectively.

  The rotation driving unit 103 rotates the rotating color filter 102 at a predetermined number of rotations in synchronization with the imaging period of the imaging element 209 based on a control signal from the control unit 302 of the control device 300. For example, if the color filters are rotated 20 times per second, each color filter crosses the incident white light at 1/60 second intervals. The imaging device 209 is, for example, a monochrome single-plate imaging device, and completes imaging and transfer of reflected light image data with each of the three primary colors (R, G, or B) at 1/60 second intervals. To do. That is, R image data, G image data, and B image data are captured in a frame sequential manner at 1/60 second intervals.

  The imaging unit 200 is formed to be elongated and bendable, for example, to enable insertion into a body cavity. The imaging unit 200 includes a light guide fiber 201 for guiding the light collected by the light source unit 100, and an illumination lens 202 that diffuses the light guided to the tip by the light guide fiber 201 and irradiates the observation target. Including. The imaging unit 200 includes an objective lens 203 that collects reflected light returning from the observation target, a variable aperture control unit 204, a variable aperture unit 205, a focal position control unit 206, and a focal position adjustment lens 207. In addition, the imaging unit 200 includes an imaging element 209 for detecting the focused imaging light, and an A / D conversion unit 210 that converts an analog signal obtained by photoelectric conversion by the imaging element 209 into a digital signal. . The image sensor 209 is configured by, for example, a CCD or a CMOS.

  The control device 300 includes an image processing unit 301 and a control unit 302. The image processing unit 301 performs image processing on the image data from the A / D conversion unit 210, performs the above-described matching processing, and performs control to display the current imaging region on the display unit 400 from the result of the matching processing. To do. The control unit 302 controls each component of the endoscope apparatus.

  The display unit 400 is a display device capable of displaying moving images, and is configured by, for example, a CRT or a liquid crystal monitor.

  The external I / F unit 500 is an interface for performing input from the user to the endoscope apparatus. The external I / F unit 500 includes, for example, a power switch for turning on / off the power, a shutter button for starting a photographing operation, and a mode switching knob for switching a photographing mode and various other modes. The mode switching knob is, for example, a zoom operation knob 501 shown in FIG. By moving the zoom operation knob 501 in the direction of the magnification observation mode, the focus position adjustment lens is moved, and a state in which the subject is in focus at a position closer to the subject is obtained. The external I / F unit 500 outputs the input information to the control unit 302.

  Next, the two observation modes of this embodiment will be described. One of the observation modes is a normal observation mode (entire observation mode) that achieves high depth and low resolution, and the other is an enlargement observation mode (close-up observation mode) that achieves low depth and high resolution.

  In order to switch between the two observation modes, the control unit 302 sends a control signal to the variable aperture control unit 204 and the focus position control unit 206 based on a mode switching request from the user using the mode switching button of the external I / F unit 500. Send. The variable aperture control unit 204 controls the aperture amount of the variable aperture unit 205 based on the received control signal. The focal position control unit 206 controls the focal position adjustment lens 207 based on the received control signal.

  When the magnification observation mode is selected, the aperture is opened to enlarge the lens aperture. Specifically, the permissible circle of confusion is set to a size that allows the limit resolution of the image sensor 209 to be obtained, and the aperture is set so that the circle of confusion at the focal point of the lens is equal to or less than the permissible circle of confusion. Further, the focal position control unit 206 drives the focal position adjustment lens so that a predetermined focal position is set in advance.

  On the other hand, when the normal observation mode is selected, the aperture is reduced to reduce the lens aperture. Specifically, the aperture is set so as to have a circle of confusion that provides practical resolution and depth of field. Further, the focal position control unit 206 drives the focal position adjustment lens so as to obtain a focal position for obtaining a practical depth of field.

  Switching between these modes is performed by first approaching the subject to the depth limit in the normal observation mode and switching to the magnification observation mode in that state. In this way, a high-definition image that is in focus can be captured even when the depth of field during close-up magnification observation is very narrow.

3. Image Processing Unit FIG. 7 shows a detailed configuration example of the image processing unit 301 that performs the above-described matching processing and the like. The image processing unit 301 includes an image configuration processing unit 310, a target magnification image group selection unit 320, a different magnification image position determination unit 330, a display information generation unit 340, and a synthesis unit 350.

  The image data output from the A / D conversion unit 210 is input to the image configuration processing unit 310. A control unit 302 is further connected to the image configuration processing unit 310, and processing parameters stored in advance in the control unit 302 are input. For example, the processing parameters are an OB clamp value, a WB coefficient, a color correction coefficient, a gradation conversion table, an edge enhancement level, and the like. Based on these processing parameters, the image configuration processing unit 310 generates a moving image that can be observed on the display unit 400 from the input image data, and outputs it to the target magnification image group selection unit 320 and the synthesis unit 350.

  A moving image output from the image configuration processing unit 310 is input to the target magnification image group selection unit 320, and an observation mode signal designated by the user in the external I / F unit 500 is input from the control unit 302. The observation mode signal is information indicating the magnification observation mode or the normal observation mode. The target magnification image group selection unit 320 receives a plurality of pieces of target magnification information to be selected that are set in advance based on the maximum magnification information of the imaging unit 200. The maximum magnification information is the image magnification at the time of the most recent focusing based on the design information of the optical system.

  The target magnification image group selection unit 320 selects representative images having a plurality of target magnifications. More specifically, the target magnification image group selection unit 320 is in an operating state in the magnification observation mode. In magnified observation, the magnification of the observation target region 10 increases as the relative distance between the observation target region 10 and the objective lens 203 approaches. For example, the target magnification image group selection unit 320 selects a representative image having a target magnification corresponding to 1, 1/4, 1/16, or 1/64 times the maximum magnification of the optical system from such a moving image. Then, the target magnification image group selection unit 320 sequentially selects the target magnification from the input moving image, overwrites and saves the selected image as the latest image of the target magnification image, the saved target magnification image and its update information. Is output to the different magnification image position determination unit 330. Details will be described later.

  Here, the reference magnification when selecting a representative image is, for example, a captured image at the time when the observation mode is switched to the magnification observation mode. The target magnification is a magnification obtained by dividing the reference image to the maximum magnification by a predetermined integer.

  The different magnification image position determination unit 330 receives a representative image of a plurality of target magnifications output from the target magnification image group selection unit 320, an observation mode signal from the control unit 302, and target magnification information. The different magnification image position determination unit 330 performs an image matching process between the input representative images of a plurality of magnifications, and calculates position information indicating where the high magnification image is located in the low magnification image. Then, the different magnification inter-image position determination unit 330 outputs the low magnification image and the position information used in the inter-image matching processing to the display information generation unit 340.

  Based on the input low-magnification image and position information, the display information generation unit 340 generates a low-magnification image representing the observation area currently being observed with a rectangular frame or the like, and outputs the generated image to the synthesis unit 350. .

  The composition unit 350 receives the low-magnification image representing the observation region from the display information generation unit 340, the moving image output from the image configuration processing unit 310, and the observation mode signal from the control unit 302. For example, when the observation mode signal indicates the magnification observation mode, the synthesis unit 350 generates a display image having a two-screen display. The combining unit 350 outputs the display image to the display unit 400.

  FIG. 8 shows an example of a two-screen display image. As shown in D1 of FIG. 8, the currently observed moving image is displayed in the display area on the left side of the screen. As shown in D2, a low-magnification image is displayed in the display area on the right side of the screen. The low-magnification image is, for example, an image with the lowest magnification among target magnification images described later. For example, these display areas are surrounded by a display frame. As indicated by D3, a frame line representing the current observation area is displayed on the low-magnification image. In addition, as indicated by D4, a frame line representing the observation target is displayed on the low-magnification image. In the normal observation mode, a display image is generated by combining the moving image indicated by D1 and the display frame.

  The composite image output from the composition unit 350 is input to the display unit 400. The input image is, for example, a high vision signal. The display unit 400 is a display device such as a liquid crystal monitor, for example, and displays an input composite image.

4). Target Magnification Image Group Selection Unit FIG. 9 shows a detailed configuration example of the target magnification image group selection unit 320. The target magnification image group selection unit 320 includes a frame memory 321, an image selection unit 322, a region extraction unit 323, a magnification correction unit 324, a target magnification image selection unit 325, a matching processing unit 326, a reference magnification image selection unit 327, and a frame memory 328. The target magnification image selection control unit 329 is included.

  The frame memory 321 stores the moving image from the image configuration processing unit 310. The moving image output from the image configuration processing unit 310 is captured while the tip of the imaging unit 200 is gradually approaching the observation target region (for example, E1 in FIG. 10). Therefore, the frame memory 321 stores a moving image that tends to increase the image magnification with time.

  For example, the frame memory 321 has a capacity capable of storing a moving image for several seconds and functions as a ring buffer. That is, the frame memory 321 sequentially overwrites an already stored image with a new image. As the moving image stored in the frame memory 321, one frame image at a predetermined position is selected by the image selection unit 322, and the selected image is output to the region extraction unit 323 and the target magnification image selection unit 325. Note that moving images stored in the frame memory 321 may be thinned at predetermined intervals. In this case, the moving image can be stored for a longer time.

  The region extraction unit 323 receives one frame image from the image selection unit 322 and extraction region information corresponding to the target magnification from the target magnification image selection control unit 329. Based on the extracted region information, the region extracting unit 323 extracts a predetermined region around the region having the highest contrast at or near the center of one frame image, and outputs the extracted region image to the magnification correcting unit 324.

  The magnification correction unit 324 receives the extracted region image output from the region extraction unit 323 and the target magnification information from the target magnification image selection control unit 329. The magnification correction unit 324 performs reduction processing on the extracted region image based on the target magnification information, and outputs the reduced extraction region image to the matching processing unit 326.

  The target magnification image selection unit 325 receives the target magnification information from the target magnification image selection control unit 329 and the matching determination information output from the matching processing unit 326. The target magnification image selection unit 325 selects a target magnification image from the frame memory 321 based on these pieces of information, and outputs the selected target magnification image to the frame memory 328. When the target magnification information is the minimum magnification, the target magnification image selection unit 325 does not depend on the information from the matching processing unit 326, and the frame image stored in the frame memory 321 at the time of switching to the magnification observation mode is The image is output to the frame memory 328 as a target magnification image having the minimum magnification. In the frame memory 328, the frame image output from the target magnification image selection unit 325 is overwritten and saved in a storage area designated for each target magnification.

  The reference magnification image selection unit 327 reads the reference magnification image from the frame memory 328 based on the target magnification information output from the target magnification image selection control unit 329, and outputs the read reference magnification image to the matching processing unit 326. The reference magnification image is an image that is matched with the reduced extraction region image in the matching processing unit 326.

  The matching processing unit 326 performs a matching process between the reference magnification image and the reduced extraction region, and outputs a correlation value at a position where the correlation is highest to the target magnification image selection unit 325. For example, the matching processing unit 326 performs enhancement processing for further emphasizing the blood vessel structure on the reference magnification image and the reduced extraction region, and then performs matching processing between the two images. For the matching process, for example, a sum of absolute differences (or a sum of squared differences) is used. The template image in the matching process is a reduced extraction area image, and the search range image in the matching process is a reference magnification image. That is, the reduced extraction area image is smaller than the reference magnification image, and the sum of absolute differences is calculated by moving the template image within the search range image. Then, the calculated minimum sum of absolute differences is output to the target magnification image selection unit 325 as a correlation value between images.

  The target magnification image selection unit 325 stores input correlation values for a plurality of frame periods. Then, the target magnification image selection unit 325 reads out a frame image corresponding to the smallest correlation value among the plurality of correlation values from the frame memory 321 as a target magnification image, and stores it in a predetermined storage area of the frame memory 328. The target magnification image selection unit 325 outputs information indicating the completion of selection of the target magnification image to the target magnification image selection control unit 329.

  The target magnification image selection control unit 329 stores update information of the target magnification image that has been selected, and outputs the update information to the different magnification image position determination unit 330. The update information of the target magnification image is stored in a format in which the magnification of the most recently updated image can be determined. For example, when there are four target magnifications to be selected, 4-bit information is prepared, and only 1 bit corresponding to the updated magnification is set as update information.

  In the magnification observation mode, the target magnification image selection unit 325 fixes the target magnification to the maximum magnification after updating up to the maximum magnification image, and thereafter continues to select the captured image as the maximum magnification target magnification image. In this case, when the magnitude of the correlation value from the matching processing unit 326 is stable and equal to or less than the predetermined threshold value, a state in which position detection (framing) can be performed by the next different magnification image inter-position determining unit 330 is possible. Continue to select. On the other hand, when the case where the correlation value is larger than the predetermined threshold continuously occurs, it is assumed that position detection is impossible, and all the bits of the update information are reset to 0, and the update information is set between different magnification images. You may output to the position determination part 330. FIG. In this case, the display information generation unit 340 generates a low-magnification image on which alert information indicating that framing is impossible is superimposed, and outputs the image to the synthesis unit 350.

  The operation of the target magnification image group selection unit will be described in detail with reference to FIGS. In the following, a case where the target magnification is 1/64, 1/16, 1/4, or 1 times the maximum magnification will be described as an example. The minimum magnification used below is the lowest magnification among the target magnifications, and is not necessarily the same as the minimum magnification originally possessed by the optical system.

  As indicated by E1 in FIG. 10, the observation target is enlarged while switching to the magnification observation mode and gradually bringing the scope closer to the observation target. Then, as indicated by E2, a target magnification image is sequentially selected from the series of captured images. For example, if the minimum magnification (reference magnification) is set to 1 and the maximum magnification is set to 64 times, the target magnifications are 1 time, 4 times, 16 times, and 64 times.

  More specifically, as shown in FIG. 11A, in the normal observation mode, the scope is brought closer to the observation object until the near point KP1 of the depth of field approaches the observation object (near point KP1 '). The imaging magnification at this time is the minimum magnification. Then, as shown in FIG. 11B, the mode is switched to the magnification observation mode, and the scope is brought closer to the observation object until the near point KP2 approaches the observation object (near point KP2 '). The imaging magnification at this time is the maximum magnification.

  Then, as shown in FIG. 12, the frame image FP2 at the time of switching to the magnification observation mode is stored as a 1 × target magnification image BP1. Next, the frame image FP3 is reduced to a quarter image size to generate a template image KP3, and the target magnification image BP1 is used as a reference magnification image and matching processing with the template image KP3 is performed. Then, the frame images after FP4 are similarly reduced and matched in the same manner, and the frame image FP4 having the highest correlation is selected as the 4 × target magnification image BP2. Next, matching processing is performed using the target magnification image BP2 as a reference magnification image, and a 16 × target magnification image is selected. In this way, the target magnification images are sequentially selected, and the target magnification image of 64 times is selected. After the 64 × target magnification image is prepared, the 64 × target magnification image is updated with the captured image.

5. Different Magnification Image Position Determination Unit FIG. 13 shows a detailed configuration example of the different magnification image position determination unit 330. The different magnification image position determination unit 330 includes a first image selection unit 331 (first magnification image selection unit), a second image selection unit 332 (second magnification image selection unit), an area extraction unit 333, a magnification correction unit 334, A matching processing unit 335 and a magnification / extraction region control unit 336 are included. Hereinafter, a case where four target magnification images BP1 to BP4 are stored in the frame memory 328 will be described as an example.

  The first image selection unit 331 and the second image selection unit 332 receive hierarchy level information for performing matching between images output from the magnification / extraction region control unit 336. As shown in FIG. 14, in the case of four types of magnification images, the hierarchy level is three. The first layer is a matching between the image BP4 with the maximum magnification currently observed and the image BP3 with the next highest magnification. The second hierarchy is matching between the image BP3 and the image BP2 having the third largest magnification. The third hierarchy is matching between the image BP2 having the third largest magnification and the image BP1 having the smallest magnification.

  Hereinafter, each component will be described by taking the processing in the first hierarchy as an example. In addition, since the same process is performed also in another hierarchy, description is abbreviate | omitted suitably about the same process.

  The first image selection unit 331 reads the currently observed image BP4 having the highest magnification from the frame memory 328 and outputs the read image BP4 to the region extraction unit 333. The second image selection unit 332 reads the image BP3 having the second largest magnification from the frame memory 328 and outputs the image BP3 to the matching processing unit 335.

  The region extraction unit 333 extracts a specified region in the input image based on the extraction region information from the magnification / extraction region control unit 336 and outputs the extracted region image to the magnification correction unit 334.

  The magnification correction unit 334 generates a reduced extraction region image obtained by reducing the extraction region image (magnification correction) based on the correction magnification information from the magnification / extraction region control unit 336, and the reduced extraction region image is matched with the matching processing unit 335. Output to.

  The matching processing unit 335 performs a matching process between the reduced extraction area image from the magnification correction unit 334 and the image BP3 output from the second magnification image selection unit. Here, the matching process uses, for example, a sum of absolute differences (or a sum of squared differences). The template image in the matching process is a reduced extraction area image corresponding to the image BP4, and the search range image in the matching process is the image BP3. That is, the reduced extraction area image is smaller than the image BP3, and the sum of absolute differences is calculated by moving the template image within the search range image. Then, the position information at the position where the calculated sum of absolute differences is the minimum value is output to the magnification / extraction area control unit 336 as matching position information in the first hierarchy.

  In the second hierarchy, the area extraction unit 333 extracts a designated area in the input image based on the matching position information calculated in the first hierarchy from the magnification / extraction area control unit 336, and the extracted area image is scaled up. The data is output to the correction unit 334. Similarly in the third layer, region extraction is performed based on the matching position information calculated in the second layer.

  In this way, the matching position information is calculated in the first to third hierarchies, and the matching position information is output to the display information generation unit 340. The maximum magnification image BP4 first stored in the frame memory 328 after switching to the magnification observation mode is considered to be an image of the target position that the user wants to observe. As the target position information, matching position information calculated in the third hierarchy for the image BP4 stored first is recorded in a memory (not shown) of the display information generation unit 340. As shown in D4 of FIG. 8, display information is displayed based on the target position information.

  In this way, it is possible to detect where the image BP4 of the region currently being magnified and observed is in the wide field image BP1 with a small magnification. At this time, it is possible to allow detection up to a deviation (displacement) within the range of the image BP3 having the second largest magnification.

  Next, the operation of the different magnification image position determination unit will be described in detail with reference to FIG. In the following, the magnifications of the images BP4 to BP1 are 64 times, 16 times, 4 times, and 1 time. First, images BP4 and BP3 are selected as targets for the first matching process. Next, the template image TP4 is generated by reducing the image BP4 to ¼ the image size. Then, matching processing between the template image TP4 and the image BP3 is performed, and an area corresponding to the template image TP4 is detected as indicated by F1.

  Next, the images BP3 and BP2 are selected as targets for the second matching process. As shown in F2, an area corresponding to the template image TP4 and its peripheral area are extracted from the image BP3. A template image TP3 is generated by reducing the extracted image to an image size of 1/4. Then, matching processing between the template image TP3 and the image BP2 is performed, and an area corresponding to the template image TP3 is detected as indicated by F3. Then, the same processing is repeated for each layer, and finally an area on the image BP1 corresponding to the image BP4 is detected.

  Now, as described above, since the field of view is narrow in magnified observation, even if a small shake occurs, the observation target deviates from the field of view, and it is necessary to repeat the operation of returning to a low magnification and enlarging the observation target again. Thus, the subject that the tolerance | permissible_range with respect to the blurring in expansion observation is narrow occurs.

  In this regard, according to the present embodiment, as shown in FIG. 7, the image processing apparatus (image processing unit 301) includes an image acquisition unit (image configuration processing unit 310) and an image selection unit (target magnification image group selection unit). 320) and an area detection unit (inter-magnification image position determination unit 330). Then, the image acquisition unit acquires a plurality of images obtained by enlarging and imaging the observation object at a plurality of different magnifications. An image selection part selects the 1st-nth magnification image (target magnification image) from which the magnification between each image differs relatively from the several images (n is a natural number of n> = 2). The region detection unit selects a reference image and a detection target image having a higher magnification than the reference image from the first to nth magnification images, and detects a region on the reference image corresponding to the detection target image. Then, the detection target image is associated with the area.

  As a result, it is possible to improve the allowable range of deviation between the scope tip and the observation target in the magnified observation. That is, according to the present embodiment, it is possible to calculate a shift amount hierarchically between a plurality of images having different magnification rates (different fields of view of captured images). Therefore, even when a deviation amount exceeding the field of view of the captured image occurs during high-magnification observation, the deviation amount from the observation target region can be calculated. This makes it possible to quickly introduce the observation target region into the field of view by confirming the display image while maintaining the magnified observation state.

  Here, the association is to associate the coordinates of the area detected on the reference image, the reference image, and the detection target image. For example, these pieces of information are stored in a memory (not shown) as a set of data. By doing so, association is performed.

  In the present embodiment, as shown in FIG. 14, the i-th magnification image in the first to n-th magnification images (BP1 to BP4) has a higher magnification than the i-1th magnification image (i is 2). ≦ i ≦ n integer). In this case, the region detection unit selects the i−1 magnification image (for example, BP1) as the reference image, and selects the i th magnification image (BP2) as the detection target image. Then, the region detection unit detects a region on the i-1th magnification image corresponding to the i-th magnification image, sequentially repeats the detection in each layer, and the first magnification image (BP4) corresponding to the nth magnification image (BP4). An area on BP1) is detected.

  In this way, the position of the observation target region on the first magnification image can be detected by repeating detection hierarchically. Further, by repeating the detection hierarchically, the matching accuracy can be improved as compared with the case where the nth magnification image and the first magnification image are directly matched.

  In the present embodiment, as illustrated in FIG. 9, the image selection unit includes a magnification correction unit 324, a matching processing unit 326, and a target magnification image selection unit 325. Then, as illustrated in FIG. 12, the magnification correction unit 324 reduces the matching target images (for example, FP3 to FP5) in the plurality of images at a predetermined magnification (1/4). The matching processing unit 326 performs matching processing between the reduced matching target image (KP3 to KP5) and the reference magnification image (BP1) in the first to nth magnification images. The target magnification image selection unit 325 selects an image (FP4) having the highest correlation in the matching process from a plurality of images (FP3 to FP5).

  In this way, the target magnification image can be selected from the captured image. Further, by reducing the matching target image at a predetermined magnification and performing matching processing, a target magnification image at a predetermined magnification interval can be selected.

  In the present embodiment, as shown in FIG. 9, the image selection unit includes a repetition control unit (target magnification image selection control unit 329). Then, the repetitive control unit selects the matching process, the process of selecting the image having the highest correlation in the matching process, and the selected image having the highest correlation as the reference magnification image used for the next matching process. Control to repeat the process.

  More specifically, the image selection unit includes a storage unit (frame memory 328) and a standard magnification image selection unit (reference magnification image selection unit 327). Then, as shown in FIG. 12, when the jth magnification image (for example, BP1) is stored in the storage unit, the reference magnification image selection unit selects the jth magnification image as the reference magnification image (j is n-1 or less natural number). The matching processing unit 326 performs matching processing using the j-th magnification image. The target magnification image selection unit 325 selects an image having the highest correlation as the kth magnification image (BP2) (k is an integer of 2 ≦ k ≦ n). The storage unit stores the kth magnification image. Then, the reference magnification image selection unit selects the kth magnification image (BP2) as the reference magnification image used for the next matching process.

  In this way, the target magnification image can be sequentially selected from the captured image by repeating the matching process. Further, the matching process can be made common by sequentially selecting the reference magnification image and the matching target image.

  In the present embodiment, as illustrated in FIG. 9, the image selection unit includes a region extraction unit 323 that extracts a matching target region from the matching target image. Then, the magnification correction unit 324 reduces the matching target region at a predetermined magnification, and the matching processing unit 326 performs matching processing between the reduced matching target region and the reference magnification image.

  In this way, it is possible to extract a region suitable for the matching process and perform the matching process. Thereby, matching accuracy can be improved. For example, if the center portion of the image is extracted, the influence of image distortion due to the optical system can be reduced.

  In the above embodiment, the case where the matching process is performed by reducing the image on the low magnification side has been described. However, in the present embodiment, the matching process may be performed by enlarging the image on the high magnification side. That is, the image processing apparatus includes a magnification correction unit that enlarges a reference magnification image (for example, BP1) in the first to nth magnification images at a predetermined magnification (4 times), the enlarged reference magnification image, and a matching target A matching processing unit that performs matching processing with the images (FP3 to FP5).

  In the present embodiment, as illustrated in FIG. 13, the region detection unit includes a detection target image selection unit (first image selection unit 331), a reference image selection unit (second image selection unit), and a magnification correction unit 334. And a matching processing unit 335. As illustrated in FIG. 15, the detection target image selection unit selects a detection target image (for example, BP4) from the first to nth magnification images (BP1 to BP4). The reference image selection unit selects a reference image (BP3) from the first to nth magnification images. The magnification correction unit reduces the detection target image to the magnification of the reference image (by 1/4). Then, the matching processing unit 335 performs matching processing between the reduced detection target image (TP4) and the reference image (BP3).

  In this way, position detection between images can be performed in each layer of the target magnification image. Further, by performing the matching process after reducing the detection target image to the magnification of the reference image, the matching process can be performed between images having different magnifications.

  In the present embodiment, as shown in FIG. 13, the region detection unit includes a repetition control unit (magnification / extraction region control unit 336). The repetitive control unit first selects a matching process, a process of selecting a reference image used for the matching process as a detection target image used for the next matching process, and a low-magnification image next to the detection target image. The process of selecting as a reference image to be used for the next matching process from the nth magnification image is repeated.

  Specifically, as illustrated in FIG. 15, the detection target image selection unit selects the mth magnification image (for example, BP4) as the detection target image. The reference image selection unit selects the p-th magnification image (BP3) having the lower magnification next to the m-th magnification image (BP4) as the reference image. The matching processing unit 335 detects a region (F1) on the pth magnification image (BP3) corresponding to the mth magnification image (BP4). Then, the detection target image selection unit selects the p-th magnification image (BP3) as the detection target image used for the next matching process. The reference image selection unit selects the qth magnification image (BP2), which is the next lower magnification than the pth magnification image (BP3), as the reference image used for the next matching process.

  In this way, it is possible to detect the current shooting area in the first magnification image by repeatedly detecting the position of each layer of the target magnification image. Further, the matching process can be made common by sequentially selecting the reference image and the detection target image.

  In the present embodiment, as shown in FIG. 13, the region detection unit includes a region extraction unit 333. Then, as illustrated in FIG. 15, the region extraction unit 333 converts the peripheral region (F2) including the region (F1) on the pth magnification image (BP3) corresponding to the mth magnification image (BP4) to the pth magnification image. Extracted from (BP3) and set as a detection target image used in the next matching process.

  In this way, it is possible to suppress the matching accuracy from being lowered due to the template image becoming too small.

  In this embodiment, as shown in FIG. 8, the display control unit (see FIG. 7) performs control to display the first magnification image (D2) and the identification information (D3) representing the area corresponding to the nth magnification image. A combining unit 350).

  In the present embodiment, the display control unit performs control to display identification information (D4) representing a target region that is an observation target in the first magnification image (D2).

  In the present embodiment, the image acquisition unit (image configuration processing unit 310) acquires a series of moving image images obtained by imaging the observation target. The display control unit (compositing unit 350) performs control to display the series of moving image images in the first display area (D1). In the second display area (D2), a moving image that has a lower magnification than the moving image displayed in the first display area, and that is selected from the series of moving images, has a lower magnification. It is displayed as an image (first magnification image). The second display area (D2) is information representing an area on the low-magnification image corresponding to the moving image displayed in the first display area (D1), and moves the viewing range of the series of moving images. The identification information (D3) that moves on the low-magnification image in response to is displayed.

  In this way, the current shooting area can be displayed on the low-magnification image. As a result, the current position of the shooting area can be known even when the shooting area is deviated from the target. Moreover, the target can be quickly returned to the field of view by displaying the target area.

6). Second Configuration Example of Endoscopic Device In the present embodiment, the scope tip may be bent using the result of the matching process described above, and the observation target may be automatically introduced into the visual field range. In the present embodiment, not only white light but also narrow band light may be taken, and matching processing may be performed using the narrow band light image.

  FIG. 16 shows a second configuration example of such an endoscope apparatus. This endoscope apparatus includes a light source unit 100, an imaging unit 200, a control device 300, a display unit 400, and an external I / F unit 500. In FIG. 4 and the like, the same components as those described above are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.

  The light source unit 100 outputs white light and narrowband light. The light source unit 100 includes a white light source 101, a rotating color filter 102, a rotation driving unit 103, and a condenser lens 104.

  FIG. 17 shows a detailed configuration example of the rotating color filter 102. The rotating color filter 102 includes a color filter 801 that shows the spectral transmittance of white light, a color filter 802 that shows a narrow-band spectral transmittance as special light, and a rotary motor 803. As shown in FIG. 18, the color filter 801 has spectral characteristics that transmit light with a wavelength of, for example, 380 nm to 650 nm. As shown in FIG. 19, the color filter 802 has spectral characteristics that transmit light having a wavelength of 390 nm to 445 nm (B2) and a wavelength of 530 nm to 550 nm (G2).

  The rotation driving unit 103 rotates the rotating color filter 102 at a predetermined number of rotations in synchronization with the imaging period of the imaging element 209 based on a control signal from the control unit 302 of the control device 300. For example, if the color filters are rotated 30 times per second, each color filter crosses incident white light at 1/60 second intervals. Then, the image sensor 209 completes imaging and transfer of image data of reflected light with respect to white light and special light (NBI) at 1/60 second intervals.

  The imaging unit 200 includes a light guide fiber 201, an illumination lens 202, an objective lens 203, a variable diaphragm control unit 204, a variable diaphragm unit 205, a focal position control unit 206, a focal position adjustment lens 207, and an imaging element. 209 and an A / D conversion unit 210.

  The image sensor 209 is an image sensor that can perform color imaging, and is configured by, for example, a Bayer array primary color single-plate image sensor. Alternatively, the imaging element 209 may be configured by a complementary color single plate imaging element.

  The control device 300 includes an image processing unit 301, a control unit 302, and a tip bending drive unit 303. In addition to the control described above with reference to FIG. 4 and the like, the control unit 302 further controls the tip bending drive unit 303 and an AF control unit 360 described later. The control unit 302 performs control according to the configuration of the rotating color filter 102.

  FIG. 20 shows a second detailed configuration example of the image processing unit 301. The image processing unit 301 includes an image configuration processing unit 310, a target magnification image group selection unit 320, a different magnification image position determination unit 330, a display information generation unit 340, a synthesis unit 350, and an AF control unit 360. The image processing unit 301 further includes an AF control unit 360 in addition to the components described above with reference to FIG.

  The AF control unit 360 performs autofocus control on the imaging unit 200. Specifically, the AF control unit 360 performs autofocus control using a contrast method or the like, and controls the focus position control unit 206 to focus the subject image.

  FIG. 21 schematically shows the tip of the imaging unit 200 (scope). The imaging unit 200 includes wires 1001 and 1003 for bending the tip in the horizontal direction (horizontal scanning direction) of the display image, and wires 1002 and 1004 for bending the tip in the vertical direction (vertical direction) of the display image. . For example, in order to bend in the left-right direction by an angle θ, the wires 1001 and 1003 are pushed and pulled by a predetermined amount N. Further, in order to bend in the vertical direction by an angle θ, the wires 1002 and 1004 are pushed and pulled by a predetermined amount N.

  FIG. 22 schematically shows the imaging timing of the white light image and the special light image. As shown in FIG. 22, the rotating color filter 102 of the light source unit 100 rotates to illuminate white light and special light (NBI) in a time-sharing manner. Then, the image sensor 209 alternately captures the white light image and the special light image, the A / D conversion unit 210 performs A / D conversion, and the image configuration processing unit 310 outputs the white light image and the special light image. The image configuration processing unit 310 outputs the special light image to the target magnification image group selection unit 320, and outputs the white light image and the special light image to the synthesis unit 350. Then, the target magnification image group selection unit 320 and the different magnification image position determination unit 330 perform matching processing using the special light image.

  In this way, matching processing can be performed using a special light image (NBI) that is an image having a higher blood vessel contrast than a white light image. Thereby, matching accuracy can be improved.

7). AF Control Unit FIG. 23 shows a detailed configuration example of the AF control unit 360. The AF control unit 360 includes an area extraction unit 361, a contrast calculation unit 362, a lens movement amount calculation unit 363, and a focused image selection unit 364.

  The area extraction unit 361 extracts a designated AF area of the special light image that is an input image based on the AF area information input from the control unit 302, and outputs the image of the designated AF area to the contrast calculation unit 362.

  The contrast calculation unit 362 calculates, for example, a difference value between the maximum value and the minimum value of the G signal in the extracted AF area as a contrast value. The calculated contrast value is stored for a plurality of frames as time-series data in a memory (not shown) of the contrast calculation unit 362. For example, the plurality of frames is at least 3 frames. The contrast values of a plurality of frames stored in the memory are output to the lens movement amount calculation unit 363.

  The lens movement amount calculation unit 363 has a memory (not shown), and the current adjustment position of the focus position adjustment lens 207 is stored in the memory. The lens movement amount calculation unit 363 calculates an adjustment position (lens adjustment position) to be a focus position from the adjustment positions associated with the input contrast values of a plurality of frames, and outputs the adjustment position to the control unit 302. To do. The control unit 302 outputs the calculated adjustment position to the focus position control unit 206. The focal position control unit 206 moves the focal position adjustment lens 207 based on the calculated adjustment position. In this way, autofocus is possible. The association between the contrast value and the lens adjustment position is realized by, for example, a lookup table that outputs the lens adjustment position with reference to the contrast value.

  The focused image selection unit 364 receives the contrast value from the contrast calculation unit. The focused image selection unit 364 determines the threshold value of the contrast value, determines whether the currently processed frame is in focus in an AF area 1100, which will be described later, and whether to output the frame image to the target magnification image group selection unit 320. Select. The focused image selection unit 364 outputs a frame image when it is determined to be in focus, and does not output a frame image when it is determined to be out of focus. That is, as the moving image input to the target magnification image group selection unit 320, only the special light image determined to be in focus via the AF control unit 360 is input instead of the image configuration processing unit 310.

  Next, the operation of the AF control unit will be described. A special light image output from the image configuration processing unit 310 is input to the AF control unit 360. As shown in FIG. 24, an AF area 1100 determined by user designation is input from the control unit 302.

  For example, the AF area 1100 is specified by the user selecting an AF area close to the observation target Tg from among a plurality of AF areas AFA. For example, when the zoom operation knob 501 shown in FIG. 10 is moved in a predetermined direction to select the close-up magnification observation mode, the AF area designation screen is displayed on the display unit 400, and the user selects the AF area. The synthesis unit 350 generates the AF area designation screen.

  The user selection may be selected and determined by moving the zoom operation knob 501 in a predetermined direction and holding it for a certain period of time, for example, by moving the AF area and stopping the holding. Alternatively, movement and selection may be performed using a cursor movement key on the keyboard. Alternatively, when the display unit 400 is a touch panel display device, the AF area may be directly designated with a finger.

8). Tip Curve Drive Unit FIG. 25 shows a detailed configuration example of the tip curve drive unit 303. The tip bending drive unit 303 includes a distance information acquisition unit 371, a deviation information acquisition unit 372, an angle calculation unit 373, a drive information output unit 374, a wire drive unit 375, and a current position information acquisition unit 376.

  The distance information acquisition unit 371 acquires distance information from the distal end of the scope along the viewpoint direction to the subject from the AF control unit 360. For example, the AF control unit 360 obtains distance information from the lens adjustment position using a lookup table or the like, and outputs the distance information to the distance information acquisition unit 371.

  The current position information acquisition unit 376 acquires the current position information from the different magnification image position determination unit 330. For example, as shown in FIG. 14, the current position information is position information of the current shooting area BP4 'in the image BP1.

  The deviation information calculation unit 372 calculates distance information (deviation information, displacement information) from the current imaging region to the observation target. Specifically, the current position information and the target position information are input to the deviation information calculation unit 372. For example, as shown in FIG. 14, the deviation information calculation unit 372 calculates the distance on the image BP1 between the region BP4 ′ and the target region Tg, and the distance on the subject from the distance and the optical image magnification. Is calculated. The optical image magnification is an image magnification at the time of photographing the image BP1, and is preset based on design information of the optical system.

  The angle calculation unit 373 calculates angle information representing the bending angle of the scope tip based on the distance information and the deviation information. Specifically, the angle between the current viewpoint direction and the viewpoint direction toward the observation target is obtained using the distance from the scope tip to the subject and the distance from the current imaging region to the observation target.

  The drive information output unit 374 outputs an adjustment amount of the length of the wire for driving the scope tip as drive information based on the angle information. Specifically, the drive information output unit 374 calculates a horizontal bending angle θh and a vertical bending angle θv necessary for returning the tip of the imaging unit 200 to the target position. The bending angle θh is an angle for moving the viewpoint direction in the horizontal scanning direction of the image, and the bending angle θv is an angle for moving the viewpoint direction in the vertical direction of the image. The drive information output unit 374 calculates push-pull amounts Nh and Nv of the four wires 1001, 1002, 1003, and 1004 shown in FIG. 21 based on the bending angles θh and θv. And the drive information output part 374 controls the motor which rotationally drives the rotating shaft of the angle knob which is not shown in figure which the wire has connected, and adjusts a wire with the said push-pull amounts Nh and Nv. In this way, it is possible to always continue framing the observation target region that the user wants to observe within the field of view.

  As described above, in the present embodiment, the matching processing unit 326 of the target magnification image group selection unit 320 has a specific anatomy as a matching target image (for example, FP3 to FP5 shown in FIG. 12) and a reference magnification image (BP1). Matching processing is performed using the enhanced image. Further, the matching processing unit 335 of the different magnification image inter-position determining unit 330 performs matching processing using an image in which a specific anatomy is emphasized as the detection target image (BP4 shown in FIG. 15) and the reference image (BP3). Do.

  Specifically, the specific anatomy is a blood vessel running structure (blood vessel running pattern). Alternatively, the specific anatomy may be a structure of the mucosal surface layer (a fine structure of the mucosal surface layer, a pit pattern).

  In this way, matching accuracy can be improved by using an image in which a specific anatomy is emphasized.

  In this embodiment, the image acquisition unit (image configuration processing unit 310 shown in FIG. 7) includes a special light image acquisition unit (not shown) that acquires a special light image including a subject image having information in a specific wavelength band. . An image in which a specific anatomy is emphasized is a special light image.

  Specifically, the specific wavelength band is a band narrower than the white wavelength band (for example, 380 nm to 650 nm) (NBI: Narrow Band Imaging). For example, the special light image is an in-vivo image obtained by copying the inside of a living body, and the specific wavelength band included in the in-vivo image is a wavelength band of a wavelength that is absorbed by hemoglobin in blood. For example, the wavelength absorbed by this hemoglobin is 390 nm to 445 nm (B2 component of narrowband light) or 530 nm to 550 nm (G2 component of narrowband light).

  Thereby, it becomes possible to observe the structure of the blood vessel located in the surface layer part and the deep part of the living body. In addition, by inputting the obtained signals to specific channels (G2 → R, B2 → G, B), lesions that are difficult to see with normal light such as squamous cell carcinoma can be displayed in brown, etc. Oversight of parts can be suppressed. Note that 390 nm to 445 nm or 530 nm to 550 nm are numbers obtained from the characteristic of being absorbed by hemoglobin and the characteristic of reaching the surface layer or the deep part of the living body, respectively. However, the wavelength band in this case is not limited to this. For example, the lower limit of the wavelength band is reduced by about 0 to 10% due to the variation factors such as the absorption by hemoglobin and the experimental results regarding the arrival of the living body on the surface layer or the deep part. It is also conceivable that the upper limit value increases by about 0 to 10%.

  In the present embodiment, as shown in FIG. 25, the endoscope apparatus includes a drive information output unit 374, a target region setting unit (control unit 302), and a current position information acquisition unit 376. Then, the target area setting unit sets a target area (Tg) to be an observation target. The current position information acquisition unit acquires position information of a region (BP4 ') on the first magnification image (BP1) corresponding to the nth magnification image (BP4) as current position information. The drive information output unit 374 outputs drive information based on the position information of the target area and the current position information.

  Specifically, the endoscope apparatus includes a first distance information acquisition unit (distance information acquisition unit 371), a second distance information calculation unit (deviation information calculation unit 372), and an angle calculation unit 373. The first distance information acquisition unit acquires first distance information representing the distance from the imaging unit to the subject. The second distance information calculation unit calculates second distance information representing the distance on the subject from the imaging area to the target area. The angle calculation unit 373 calculates angle information representing the angle θ from the first and second distance information. The drive information output unit 374 outputs drive information based on the angle information.

  More specifically, as illustrated in FIG. 21, the imaging unit 200 is an endoscope scope having wires 1001 to 1004. In this case, the drive information is information indicating the adjustment amount of the length of the wires 1001 to 1004. Then, the visual field range of the imaging unit 200 is moved by adjusting the length of the wires 1001 to 1004 based on the adjustment amount.

  In this way, the scope tip can be bent in the direction of the target even when the imaging region of the scope is displaced from the target. As a result, the target can be automatically introduced into the visual field range.

  In the present embodiment, as shown in FIG. 16, the imaging unit 200 includes an autofocus unit (focus position control unit 206). In addition, the imaging unit 200 includes a variable diaphragm unit 205 and a variable diaphragm control unit 204 that performs control to narrow down the variable diaphragm unit 205 as the magnification decreases when the observation object is enlarged and imaged at a plurality of different magnifications. Including. In addition, the variable aperture control unit 204 performs control to open the variable aperture unit 205 when the imaging magnification of the imaging unit 200 is equal to or greater than a predetermined magnification (enlarged shooting mode).

  In this way, since the focus can be automatically achieved by AF, the lesion diagnosis time can be shortened. Further, by performing aperture control, it is possible to take images with pan focus in the normal observation mode and with high resolution in the enlargement observation mode.

9. Software In the above-described embodiment, each unit configuring the image processing unit 301 is configured by hardware. However, the present invention is not limited to this. For example, the CPU may be configured to process each unit for an image acquired by the imaging apparatus, and may be realized as software by the CPU executing a program. Alternatively, a part of processing performed by each unit may be configured by software.

  When the processing performed by each unit of the image processing unit 301 is realized as software, for example, a known computer system such as a workstation or a personal computer can be used as the image processing apparatus. Then, a program (image processing program) for realizing the processing performed by each unit of the image processing unit 301 is prepared in advance, and this image processing program is executed by the CPU of the computer system.

  FIG. 26 is a system configuration diagram showing a configuration of a computer system 600 in the present modification, and FIG. 27 is a block diagram showing a configuration of a main body 610 in the computer system 600. As shown in FIG. 26, the computer system 600 includes a main body 610, a display 620 for displaying information such as an image on the display screen 621 according to an instruction from the main body 610, and various information on the computer system 600. A keyboard 630 for inputting and a mouse 640 for designating an arbitrary position on the display screen 621 of the display 620 are provided.

  Further, as shown in FIG. 27, a main body 610 in the computer system 600 includes a CPU 611, a RAM 612, a ROM 613, a hard disk drive (HDD) 614, a CD-ROM drive 615 that accepts a CD-ROM 660, and a USB memory. USB port 616 to which 670 is detachably connected, I / O interface 617 to which display 620, keyboard 630 and mouse 640 are connected, and a LAN interface for connection to a local area network or wide area network (LAN / WAN) N1 618.

  Further, the computer system 600 is connected to a modem 650 for connecting to a public line N3 such as the Internet, and is another computer system via a LAN interface 618 and a local area network or a wide area network N1. A personal computer (PC) 681, a server 682, a printer 683, and the like are connected.

  The computer system 600 reads out and executes an image processing program for realizing a processing procedure to be described later with reference to an image processing program (for example, FIGS. 28 to 30) recorded on a predetermined recording medium. An image processing apparatus is realized with this. Here, the predetermined recording medium is a “portable physical medium” including an MO disk, a DVD disk, a flexible disk (FD), a magneto-optical disk, an IC card, etc. in addition to the CD-ROM 660 and the USB memory 670, a computer A “fixed physical medium” such as HDD 614, RAM 612, and ROM 613 provided inside and outside the system 600, a public line N3 connected via the modem 650, and another computer system (PC) 681 or server 682 are connected. It includes any recording medium that records an image processing program readable by the computer system 600, such as a “communication medium” that stores the program in a short time when transmitting the program, such as a local area network or a wide area network N1.

  That is, the image processing program is recorded on a recording medium such as “portable physical medium”, “fixed physical medium”, and “communication medium” in a computer-readable manner. An image processing apparatus is realized by reading and executing an image processing program from a medium. Note that the image processing program is not limited to be executed by the computer system 600, and when the other computer system (PC) 681 or the server 682 executes the image processing program or in cooperation therewith. The present invention can be similarly applied to a case where an image processing program is executed.

  As an example of a case where a part of processing performed by each unit is configured by software, a processing procedure when the processing of the image processing unit 301 is realized by software for an image acquired by the imaging apparatus is illustrated in FIGS. It demonstrates using the flowchart of these. The image acquired by the imaging device is an image output by the A / D conversion unit 210, for example.

  As shown in FIG. 28, when this process is started, a captured image is acquired (step S1), and a process of selecting a target magnification image is performed (step S2). Next, a matching process between target magnification images is performed, and a process of detecting the position of the image with the maximum magnification in the 1x image is performed (step S3). Next, control for displaying the observation position is performed (step S4), and control for driving the scope tip is performed (step S5). Next, when the magnification observation mode ends, the process ends (step S6, YES). If the magnification observation mode is continued (step S6, NO), the maximum magnification image is updated (step S7), and steps S3 to S6 are repeated.

  FIG. 29 shows a detailed flowchart of the target magnification image selection process. When this process is started, a target magnification image of 1 is acquired from the captured image (step S20). Next, a first image is selected from the target magnification images (step S21). The first image is a reference magnification image serving as a reference for matching processing. Next, a second image is selected from the captured image (step S22), region extraction is performed (step S23), and the image size of the extracted region is corrected for magnification to generate a template image (step S24). Next, the reference magnification image and the template image are matched (step S25). If the correlation value is not the minimum value (step S26, NO), steps S22 to S25 are repeated. When the correlation value becomes the minimum value (step S26, YES), the second image is selected as the target magnification image (step S27). If selection to the maximum magnification target magnification image has not been completed (step S28, NO), steps S21 to S27 are repeated. When the selection up to the maximum magnification target magnification image is completed, the process is terminated (YES in step S28).

  FIG. 30 shows a detailed flowchart of the area detection process. When this process is started, a reference image is selected from the target magnification image (step S40), and a detection target image is selected from the target magnification image (step S41). Next, a region is extracted from the detection target image (step S42), and the template image is generated by correcting the magnification of the image size of the extracted region (step S43). Next, the reference image and the template image are matched (step S44). If the matching process has not been completed up to the target magnification image having the maximum magnification (step S45, NO), steps S40 to S44 are repeated. When the matching process is completed up to the maximum magnification target magnification image, the process ends (step 45, YES).

  In addition, this embodiment is a program code that realizes each part of the present embodiment (an image configuration processing unit, a target magnification image group selection unit, a position determination unit between different magnification images, a display information generation unit, a synthesis unit, an AF control unit, etc.) It can also be applied to computer program products in which is recorded.

  The computer program product includes, for example, an information storage medium (an optical disk medium such as a DVD, a hard disk medium, a memory medium, etc.) on which the program code is recorded, a computer on which the program code is recorded, and an Internet system (for example, A system including a server and a client terminal), etc., such as an information storage medium, apparatus, device or system in which a program code is incorporated. In this case, each component and each processing process of this embodiment are mounted by each module, and the program code constituted by these mounted modules is recorded in the computer program product.

  As mentioned above, although embodiment and its modification which applied this invention were described, this invention is not limited to each embodiment and its modification as it is, and in the range which does not deviate from the summary of invention in an implementation stage. The component can be modified and embodied. Further, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments and modifications. For example, some constituent elements may be deleted from all the constituent elements described in each embodiment or modification. Furthermore, you may combine suitably the component demonstrated in different embodiment and modification. Thus, various modifications and applications are possible without departing from the spirit of the invention.

  In the specification or drawings, terms (scope, narrowband light, displacement, etc.) described at least once together with different terms (imaging unit, special light, displacement, etc.) having a broader meaning or the same meaning are used in the specification or drawings. It can be replaced by the different terms at any point.

10 observation target region, 100 light source unit, 101 white light source, 102 rotating color filter,
103 rotational drive unit, 104 condenser lens, 200 imaging unit,
201 light guide fiber, 202 illumination lens, 203 objective lens,
204 variable aperture control unit, 205 variable aperture unit, 206 focus position control unit,
207 focal position adjustment lens, 209 imaging device, 210 A / D conversion unit,
300 control device, 301 image processing unit, 302 control unit,
303 tip bending drive unit, 310 image configuration processing unit, 320 target magnification image group selection unit,
321 frame memory, 322 image selection unit, 323 region extraction unit,
324 magnification correction unit, 325 target magnification image selection unit, 326 matching processing unit,
327 reference magnification image selection unit, 328 frame memory,
329 target magnification image selection control unit, 330 different magnification image position determination unit,
331 first image selection unit, 332 second image selection unit, 333 region extraction unit,
334 magnification correction unit, 335 matching processing unit, 336 magnification / extraction area control unit,
340 display information generation unit, 350 synthesis unit, 360 AF control unit,
361 region extraction unit, 362 contrast calculation unit, 363 lens movement amount calculation unit,
364 focused image selection unit, 371 distance information acquisition unit, 372 deviation information calculation unit,
373 angle calculation unit, 374 drive information output unit, 375 wire drive unit,
376 Current position information acquisition unit, 400 display unit, 500 external I / F unit,
501 Zoom operation knob, 600 computer system,
601 to 603 color filters, 610 main body, 611 CPU, 612 RAM,
613 ROM, 614 HDD, 615 CD-ROM drive,
616 USB port, 617 I / O interface,
618 LAN interface, 620 display, 621 display screen,
630 keyboard, 640 mouse, 650 modem, 660 CD-ROM,
670 USB memory, 681 PC, 682 server, 683 printer,
801, 802 color filter, 803 rotary motor, 1001-1004 wire,
AFA AF area, FP2 frame image, KP3 template image,
BP1 to BP4 target magnification image, TP4 template image, Tg observation object,
θ angle, N1 wide area network, N3 public line

Claims (39)

An image acquisition unit for acquiring a plurality of images obtained by enlarging and imaging an observation object at a plurality of different magnifications;
An image selection unit that selects from the plurality of images first to nth magnification images (n is a natural number of n ≧ 2) having relatively different magnifications between the images;
From the first to n-th magnification images, a reference image and a detection target image having a higher magnification than the reference image are selected, a region on the reference image corresponding to the detection target image is detected, and the detection target An area detector for associating an image with the area;
An image processing apparatus comprising: In claim 1,
When the i-th magnification image in the first to n-th magnification images is an image having a higher magnification than the i-th magnification image in the first to n-th magnification images,
The region detection unit
The i-1 magnification image is selected as the reference image, the i magnification image is selected as the detection target image, and regions on the i-1 magnification image corresponding to the i magnification image are sequentially detected. Thus, an area on the first magnification image corresponding to the nth magnification image is detected. In claim 1,
The image selection unit
A magnification correction unit that reduces a matching target image in the plurality of images at a predetermined magnification;
A matching processing unit that performs a matching process between the reduced matching target image and a reference magnification image in the first to nth magnification images;
A target magnification image selection unit for selecting an image having the highest correlation in the matching process from the plurality of images;
An image processing apparatus comprising: In claim 3,
The image selection unit
Control for repeating the matching process, a process for selecting an image having the highest correlation in the matching process, and a process for selecting the selected image having the highest correlation as a reference magnification image used for the next matching process An image processing apparatus having a repetitive control unit for performing the above. In claim 3,
The image selection unit
A storage unit for storing the first to n-th magnification images;
A reference magnification image selection unit for selecting the reference magnification image;
Have
The reference magnification image selection unit includes:
When the jth magnification image among the first to nth magnification images is stored in the storage unit, the jth magnification image is selected as the reference magnification image,
The matching processing unit
Performing the matching process using the j-th magnification image;
The target magnification image selection unit
Selecting the image having the highest correlation as the kth magnification image among the first to nth magnification images;
The storage unit
An image processing apparatus for storing the k-th magnification image. In claim 5,
The reference magnification image selection unit includes:
The image processing apparatus, wherein the k-th magnification image is selected as the reference magnification image used for the next matching process. In claim 3,
The image selection unit
An area extracting unit for extracting a matching target area from the matching target image;
The magnification correction unit
Reducing the matching target area at the predetermined magnification;
The matching processing unit
An image processing apparatus that performs a matching process between the reduced matching target area and the reference magnification image. In claim 3,
The matching processing unit
An image processing apparatus that performs the matching process using an image in which a specific biological structure is emphasized as the matching target image and the reference magnification image. In claim 8,
The specific anatomy is:
An image processing apparatus having a blood vessel traveling structure. In claim 8,
The specific anatomy is:
An image processing apparatus having a structure of a mucous membrane surface layer. In claim 8,
The image acquisition unit
A special light image acquisition unit that acquires a special light image including a subject image having information in a specific wavelength band;
The image in which the specific anatomy is emphasized is
An image processing apparatus characterized by being the special light image In claim 11,
The specific wavelength band is
An image processing apparatus having a narrower band than a white wavelength band. In claim 12,
The special light image is an in-vivo image of the living body,
The image processing apparatus according to claim 1, wherein the specific wavelength band included in the in-vivo image is a wavelength band of a wavelength absorbed by hemoglobin in blood. In claim 13,
The image processing apparatus according to claim 1, wherein the specific wavelength band is 390 nm to 445 nm or 530 nm to 550 nm. In claim 1,
The image selection unit
A magnification correction unit for enlarging a reference magnification image among the first to n-th magnification images at a predetermined magnification;
A matching processing unit that performs a matching process between the enlarged reference magnification image and a matching target image in the plurality of images;
A target magnification image selection unit for selecting an image having the highest correlation in the matching process from the plurality of images;
An image processing apparatus comprising: In claim 1,
The region detection unit
A detection target image selection unit that selects the detection target image from the first to n-th magnification images;
A reference image selection unit for selecting the reference image from the first to n-th magnification images;
A magnification correction unit that reduces the detection target image to the magnification of the reference image;
A matching processing unit that performs a matching process between the reduced detection target image and the reference image;
An image processing apparatus comprising: In claim 16,
The region detection unit
The matching process, the process of selecting the reference image used in the matching process as a detection target image used in the next matching process, and the next lower magnification image of the detection target image used in the next matching process An image processing apparatus comprising: a repetitive control unit that performs repetitive control of selecting a reference image to be used as a reference image to be used for a next matching process from the first to nth magnification images. In claim 16,
The detection target image selection unit includes:
Selecting the m-th magnification image among the first to n-th magnification images as the detection target image;
The reference image selection unit
As the reference image, a low-magnification p-th magnification image next to the m-th magnification image is selected from the first to n-th magnification images,
The matching processing unit
An image processing apparatus for detecting an area on the p-th magnification image corresponding to the m-th magnification image. In claim 18,
The detection target image selection unit includes:
As the detection target image used for the next matching process, the p-th magnification image is selected,
The reference image selection unit
Image processing characterized in that, as a reference image used in the next matching processing, a q-th magnification image that is the next lower magnification than the p-th magnification image is selected from the first to n-th magnification images. apparatus. In claim 18,
The region detection unit
A region extracting unit configured to extract a peripheral region including a region on the p-th magnification image corresponding to the m-th magnification image from the p-th magnification image and set as a detection target image used for a next matching process; An image processing apparatus. In claim 16,
The matching processing unit
An image processing apparatus that performs the matching process using an image in which a specific biological structure is emphasized as the detection target image and the reference image. In claim 21,
The specific anatomy is:
An image processing apparatus having a blood vessel traveling structure. In claim 21,
The specific anatomy is:
An image processing apparatus having a structure of a mucous membrane surface layer. In claim 21,
The image acquisition unit
A special light image acquisition unit that acquires a special light image including a subject image having information in a specific wavelength band;
The image in which the specific anatomy is emphasized is
An image processing apparatus characterized by being the special light image In claim 24,
The specific wavelength band is
An image processing apparatus having a narrower band than a white wavelength band. In claim 25,
The special light image is an in-vivo image of the living body,
The image processing apparatus according to claim 1, wherein the specific wavelength band included in the in-vivo image is a wavelength band of a wavelength absorbed by hemoglobin in blood. In claim 26,
The image processing apparatus according to claim 1, wherein the specific wavelength band is 390 nm to 445 nm or 530 nm to 550 nm.   An endoscope apparatus comprising the image processing apparatus according to claim 1. In claim 28,
Including a display controller,
When the i-th magnification image in the first to n-th magnification images is an image having a higher magnification than the i-th magnification image in the first to n-th magnification images,
The region detection unit
Sequentially detecting a region on the i-1 magnification image corresponding to the i magnification image, and detecting a region on the first magnification image corresponding to the n magnification image;
The display control unit
An endoscope apparatus that performs control to display identification information representing an area corresponding to the first magnification image and the nth magnification image. In claim 29,
The display control unit
An endoscope apparatus that performs control to display identification information representing a target region that is an observation target in the first magnification image. In claim 28,
A drive information output unit;
A target area setting section;
A current location information acquisition unit;
Including
When the i-th magnification image in the first to n-th magnification images is an image having a higher magnification than the i-th magnification image in the first to n-th magnification images,
The region detection unit
By sequentially detecting a region on the i-1th magnification image corresponding to the ith magnification image, a region on the first magnification image corresponding to the nth magnification image is detected,
The target area setting unit
Setting a target area to be an observation target in the first magnification image;
The current position information acquisition unit
Obtaining position information of a region on the first magnification image corresponding to the nth magnification image as current position information;
The drive information output unit
An endoscope apparatus that outputs driving information for performing driving for putting the target area into a visual field range of an imaging unit based on the position information of the target area and the current position information. In claim 31,
A first distance information acquisition unit for acquiring first distance information representing a distance from the imaging unit to the observation object;
Second distance information calculation for calculating second distance information representing a distance from the imaging area of the imaging unit to the target area from the position information of the target area, the current position information, and the magnification of the first magnification image. And
An angle calculation unit that calculates angle information representing an angle between a visual field direction of the imaging unit and a direction toward the target region from the first distance information and the second distance information;
Including
The drive information output unit
An endoscope apparatus that outputs the driving information based on the angle information. In claim 31,
The imaging unit
An endoscope scope having a wire for moving the visual field range;
The drive information is
It is information indicating the adjustment amount of the length of the wire,
The visual field range is
An endoscope apparatus that is moved by adjusting the length of the wire based on the adjustment amount. In claim 28,
An endoscope apparatus including an imaging unit having an autofocus unit. In claim 28,
An imaging unit having a variable aperture unit;
When enlarging and imaging the observation object at a plurality of different magnifications, a variable aperture control unit that performs control to reduce the variable aperture unit as the magnification is low,
An endoscopic device comprising: In claim 28,
An imaging unit having a variable aperture unit;
A variable aperture control unit that performs control to open the variable aperture unit when an imaging magnification of the imaging unit is equal to or greater than a predetermined magnification; and
An endoscopic device comprising: Obtain multiple images obtained by magnifying and observing the observation object at multiple different magnifications,
From the plurality of images, first to nth magnification images (n is a natural number of n ≧ 2) having relatively different magnifications for each image are selected,
From the first to nth magnification images, select a reference image and a detection target image having a higher magnification than the reference image,
An image processing method comprising: detecting a region on the reference image corresponding to the detection target image, and associating the detection target image with the region. An image acquisition unit for acquiring a plurality of images obtained by enlarging and imaging an observation object at a plurality of different magnifications;
An image selection unit for selecting, from among the plurality of images, first to nth magnification images (n is a natural number of n ≧ 3) having relatively different magnifications in each image;
From the first to n-th magnification images, a reference image and a detection target image having a higher magnification than the reference image are selected, a region on the reference image corresponding to the detection target image is detected, and the detection target As an area detector that associates an image with the area,
A program characterized by causing a computer to function. An image acquisition unit for acquiring a series of moving image images obtained by imaging the observation object;
A display control unit that performs control to display the series of moving image images in the first display area;
Including
The display control unit
Control for displaying a moving image selected from the series of moving images as a low-magnification image in the second display region, which is a lower magnification image than the moving image displayed in the first display region. And
The display control unit
Information representing an area on the low-magnification image corresponding to the moving image displayed in the first display area, and moves on the low-magnification image in accordance with movement of the visual field range of the series of moving image. An endoscope apparatus that controls to display identification information in the second display area.