JP2012042524A - Image processing device and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily allow three-dimensional observation of a target object.SOLUTION: An image acquisition part 311 acquires observation images of a right and left pair photographed by an imaging device arranged on right and left light paths of a stereoscopic microscope for magnification observation of a target object, a depth amount calculation part 312 calculates a depth amount on each corresponding position of the observation images of the right and left pair, and a three-dimensional observation image generation part 313 generates a three-dimensional model, based on the depth amount calculated for each corresponding position. A three-dimensional observation image generation part 314 generates a three-dimensional observation image based on the generated three-dimensional model and an image output part 315 outputs the generated three-dimensional observation image to a display device so that the three-dimensional observation image is displayed. This invention can be applied, for example, for an image processing device which performs image processing for a target image acquired from a stereoscopic microscope.

Description

  The present invention relates to an image processing apparatus and method.

  Conventionally, a confocal microscope is used for three-dimensional observation of an observation object. The confocal microscope collects the light reflected by the observation object when the laser beam is scanned two-dimensionally on the observation object, and detects the intensity of the light passing through the pinhole. A two-dimensional image having a very shallow depth of focus is generated as surface information of the observation object. Then, a three-dimensional image is generated by superimposing two-dimensional images of a plurality of focal planes obtained by repeating the operation a plurality of times in the Z-axis direction.

  That is, in the confocal microscope, in order to obtain information on the height of the observation object, two-dimensional images are sequentially acquired while moving the observation position of the observation object in the Z-axis direction, and based on these two-dimensional images. Then, the position of the observation object is detected when the intensity of the light passing through the pinhole is maximized at each point of the observation object (see, for example, Patent Document 1).

JP 2007-102102 A

  However, in the confocal microscope, in order to obtain information on the height of the observation object, it is necessary to acquire a plurality of two-dimensional images while moving the observation position in the Z-axis direction, so that a three-dimensional image is generated. There has been a demand to reduce the processing load at the time of acquiring a two-dimensional image as a source for performing the three-dimensional observation of the observation object more easily.

  The present invention has been made in view of such a situation, and by simply acquiring a pair of left and right observation images acquired from a stereomicroscope at a time, three-dimensional observation of an observation target can be performed easily. It is something that can be done.

  The image processing apparatus of the present invention includes an image acquisition unit that acquires a pair of left and right observation images captured by an imaging unit that is arranged on the left and right eye optical paths of a stereomicroscope that magnifies an observation target, and the acquired left and right pair Based on the observation image, the calculation means for calculating the depth amount with respect to the central axis of the stereomicroscope at each corresponding position of the pair of left and right observation images, and the depth amount calculated for each corresponding position 3D model generation means for generating a 3D model corresponding to the observation object, and a 3D observation image which is an image for 3D observation of the observation object based on the generated 3D model And a three-dimensional observation image generating means for generating

  According to the present invention, three-dimensional observation of an observation object can be performed easily.

It is an external view which shows one Embodiment of the stereomicroscope to which this invention is applied. It is a figure which shows the example of the illumination method of a stereomicroscope. It is a figure which shows 1st Embodiment of the microscope system to which this invention is applied. It is a figure which shows the functional structural example of a control box. It is a flowchart explaining the production | generation process of a three-dimensional observation image. It is a figure explaining the calculation method of depth amount. It is a figure which shows the example of calculation of depth amount. It is a figure which shows 2nd Embodiment of the microscope system to which this invention is applied. It is a figure which shows 3rd Embodiment of the microscope system to which this invention is applied. It is a figure which shows 4th Embodiment of the microscope system to which this invention is applied. It is a figure which shows the other example of the illumination method of a stereomicroscope.

Hereinafter, modes for carrying out the present invention (hereinafter referred to as embodiments) will be described. The description will be given in the following order.
1. First Embodiment 2. FIG. Second Embodiment 3. FIG. Third embodiment 4. 4. Fourth embodiment Modified example

<1. First Embodiment>
First, a first embodiment of the present invention will be described with reference to FIGS.

[Configuration example of stereo microscope]
FIG. 1 is an external view of an embodiment of a stereomicroscope to which the present invention is applied.

  The stereomicroscope 101 is a single-objective binocular parallel optical system (Galileo type) stereomicroscope. The stereomicroscope 101 includes a base unit (illumination unit) 111, a variable magnification lens barrel 112, and a focusing device 113. The focusing device 113 is erected on the rear end of the upper surface of the base portion 111 and at the center in the left-right direction, and the variable magnification lens barrel 112 is provided on the upper portion of the front surface of the focusing device 113.

  The base unit 111 includes a transmission illumination device (not shown), and an observation target mounting table 114 in which a transparent member is embedded is provided on the upper surface. An observation target object (observation target object 102 in FIGS. 2 and 3 to be described later) is placed on the observation target mounting table 114.

  An objective lens 115 is provided on the lower surface of the zoom lens barrel 112. As shown in FIG. 1, the objective lens 115 may have only one objective lens or a plurality of objective lenses. Then, the observer uses the stereomicroscope 101 by selecting, for example, a number of objective lenses 115 that can be attached or less from a plurality of predetermined low magnification objective lenses and a plurality of high magnification objective lenses. To do.

  A zooming knob 116 is provided on the left side surface and the right side surface of the zoom lens barrel 112. Furthermore, it is possible to attach left and right eye photographing units (photographing units 161L and 161R in FIG. 3 and the like) to the variable magnification lens barrel 112.

  A binocular tube 117 is provided above the variable magnification lens barrel 112. The binocular tube 117 is attached with an eyepiece lens 117a for the left eye, an eyepiece lens 117b for the right eye, and the like. The observer views the observation image of the left eye and the right eye through the eyepieces with the left eye and the right eye, respectively, so that the image of the observation object placed on the observation object mounting table 114 is three-dimensional. Can be seen.

  A focusing knob 118 is provided at the upper part of the left and right side surfaces of the focusing device 113. Further, the focusing device 113 is provided with a mechanism (not shown) that moves the zoom lens barrel 112 up and down along the direction perpendicular to the observation target mounting table 114 as the focusing knob 118 rotates. Yes.

[Example of lighting method]
FIG. 2 shows an example of an illumination method of the stereomicroscope 101. This example is one of the illumination methods used when the surface of the observation object 102 is opaque, such as a printed board.

  The light source device 201L and the light source device 201R are light source devices that can independently change the light amount and the spectral distribution. The illumination light emitted from the light source device 201L is guided by the light guide bundle fiber 202L, and is irradiated on the observation object 102 from the diagonally upper left direction. The illumination light emitted from the light source device 201R is guided by the light guide bundle fiber 202R, and is irradiated onto the observation object 102 from the upper right direction. By irradiating the observation object 102 with illumination light from different directions on the left and right in this way, it is possible to obtain an observation image and an observation image with good color reproducibility and appropriate contrast due to shadows on the uneven portions. .

[Configuration example of microscope system]
FIG. 3 shows a configuration example of a microscope system including the stereoscopic microscope 101 described with reference to FIGS. 1 and 2, a control box 301 for controlling the stereoscopic microscope 101, and a 2D display device 303. As shown in FIG. 3, a left-eye imaging unit 161L and a right-eye imaging unit 161R are attached to the stereomicroscope 101 that magnifies the observation object 102, and the control box 301 is connected by cables 162L and 162R. Connected with.

  In the stereomicroscope 101, light from the observation object 102 (hereinafter referred to as observation light) passes through the objective lens 151 and then enters the afocal zoom system 152L and the afocal zoom system 152R depending on the optical path. Each of the afocal zoom system 152L and the afocal zoom system 152R includes an aperture stop and a variable power lens group, and each variable power lens group includes a movable group for adjusting the observation magnification. That is, by rotating the zoom knob 116 provided on the left side surface and the right side surface of the zoom lens barrel 112 in FIG. 1, each movable group moves in the optical axis direction by a distance corresponding to the rotation amount, and observation is performed. The magnification can be adjusted. Further, the aperture amount of the aperture stop can be adjusted by an adjustment mechanism (not shown) provided in the variable magnification lens barrel 112 of FIG.

  The observation light emitted from the afocal zoom system 152L enters the imaging lens 153L and forms an image on the imaging surface of the imaging element 171L of the imaging unit 161L. Similarly, the observation light emitted from the afocal zoom system 152R enters the imaging lens 153R and forms an image on the imaging surface of the imaging element 171R of the imaging unit 161R.

  The image sensor 171L of the imaging unit 161L captures an image of the observation light imaged by the imaging lens 153L, and an image signal indicating the obtained image for the left eye (hereinafter referred to as an observation image) is transmitted to the cable. It is supplied to the control box 301 via 162L. Similarly, the image sensor 171R of the imaging unit 161R captures an image of the observation light imaged by the imaging lens 153R and transmits a signal indicating the obtained right-eye observation image via the cable 162R to the control box. 301 is supplied.

  The control box 301 controls the stereomicroscope 101, and based on image signals corresponding to the left observation image 401L and the right observation image 401R supplied from the stereomicroscope 101, an image for three-dimensional observation of the observation object 102 ( (Hereinafter referred to as “three-dimensional observation image”) is generated and displayed on the 2D display device 303. The control box 301 is configured by, for example, a computer or a processor such as a CPU (Central Processing Unit) and the like, and by executing a predetermined control program, an image acquisition unit 311 and a depth amount calculation as shown in FIG. Functions including the unit 312, the 3D model generation unit 313, the 3D observation image generation unit 314, the image output unit 315, and the CAD data conversion unit 316 are realized. The details of a series of processing examples of the image acquisition unit 311 to the image output unit 315 in FIG. 4 will be described later with reference to the flowchart in FIG. Details of the processing example of the CAD data conversion unit 316 will be described later in “5. Modifications”.

  Returning to FIG. 3, the input device 302 is a device that receives an instruction from an observer represented by, for example, a mouse or a keyboard. The input device 302 supplies the control box 301 with an operation signal corresponding to the operation of the observer, such as selection of the content of the processing executed by the control box 301 and an instruction to adjust parameters relating to image processing.

  The 2D display device 303 is a flat image display device that displays an image in a planar manner. The 2D display device 303 displays an image based on the data of the three-dimensional observation image output from the control box 301 in a bird's eye view. Thereby, the observer does not look into the eyepiece lens attached to the binocular tube 117 of the stereomicroscope 101 and observe it, and the 3D observation image displayed on the 2D display device 303 (for example, the 2D display device of FIG. 3). By viewing the triangular prism (three-dimensional image displayed on the screen 303), it is possible to observe the image of the observation object 102 having the shape shown in the shape reference diagram 400 of FIG. The shape reference diagram 400 also shows an arrow view when viewed from the direction A along the central axis of the stereomicroscope 101, in addition to the shape of the triangular prism of the observation object 102.

[Control box processing details]
Next, with reference to the flowchart in FIG. 5, the three-dimensional observation image generation process executed by the control box 301 in FIG. 4 will be described.

  In step S11, the image acquisition unit 311 acquires a pair of left and right observation images 401L and 401R captured by the imaging elements 171L and 171R arranged on the optical paths of the left and right eyes of the stereomicroscope 101, and the depth. This is supplied to the quantity calculation unit 312 and the three-dimensional model generation unit 313.

  In step S12, the depth amount calculation unit 312 corresponds to the pair of left and right left observation images 401L and right observation image 401R based on the pair of left and right left observation images 401L and right observation image 401R acquired by the image acquisition unit 311. The amount of depth at each position is obtained.

  Here, a depth amount calculation method obtained by the depth amount calculation unit 312 will be described with reference to FIG.

  In FIG. 6, the left observation image 401 </ b> L captured by the imaging device 171 </ b> L on the left eye optical path among the pair of left and right observation images acquired by the image acquisition unit 311 is illustrated in FIG. A right observation image 401R captured by the element 171R is shown in FIG. 6b.

  The depth amount calculation unit 312 determines corresponding points for associating feature points having similar characteristics in the left observation image 401L and the right observation image 401R, and the corresponding points with respect to the central axis of the stereomicroscope 101 are determined. The depth amount is obtained. As the method, a stereo matching method which is one of three-dimensional shape restoration methods can be used.

  That is, the stereo matching method is used to determine which part of the right observation image 401R corresponds to the left observation image 401L by area correlation calculation, and the depth amount is determined by using the principle of triangulation based on the correspondence relationship. It becomes possible.

  Here, the area correlation is a comparison between a predetermined region including the periphery of the pixel of interest, a difference is obtained in the left and right images for each point in the region, and the smallest sum is the corresponding point. It is a method. Specifically, as shown in FIG. 6, the luminance Il (x, y) of a point Al (x, y) at the left observation image 401L and only d in the X direction from the coordinates of the point Al in the right observation image 401R. The luminance Ir (x−d, y) at the shifted point Ar (x−d, y) is compared. That is, by finding d = dmin where the value of Il (x, y) -Ir (x-d, y) is the smallest value, the corresponding point of Al (x, y) becomes the point Ar (x-dmin , Y).

  In this example, it is a comparison of only one point, but if it is expanded to a 2m × 2n region, the following equation (1) is calculated to calculate the difference in luminance value within the plane of the predetermined region. It becomes possible to take.

  By calculating the equation (1), it is possible to obtain a difference in luminance value within a predetermined area, so that it is possible to avoid failure of the calculation even if there is occlusion. The occlusion is a state in which, when observing a three-dimensional structure, there is a portion that cannot be seen with both eyes and is visible only to one eye. For example, when there are multiple objects and the object in the foreground hides the object behind it, the part hidden by the left and right eyes may shift, or the inclined surface of the three-dimensional structure can be seen by the right eye and the left eye Occurs in situations where you cannot see.

  Subsequently, the position is detected. Here, a method using the principle of triangulation will be described. The angle at which the image sensors 171L and 171R (hereinafter also referred to as cameras L and R) arranged on the left and right eye optical paths look at a point (object point) on the observation object 102 is the inclination of the optical axis of the stereomicroscope 101. For example, when the distance between the left and right eye optical paths is D = 22 and the focal length of the objective lens 151 is fo = 100 mm, the angle (convergence angle) θ for viewing the object plane on the optical axis with the left and right cameras L and R is a half angle. Thus, it can be obtained by the following equation (2).

  θ = arctan (D / (2 × fo)) = arctan (22 / (2 × 100)) = 6.3 ° (2)

  Then, on the left eye optical path, the opening angle of the optical axes of the left and right cameras L and R is 2θ, and the coordinates of the point Al on the left observation image 401L and the corresponding point Ar on the right observation image 401R. The coordinates (x, z) of the object point A when the arranged camera L is used as a reference can be obtained. The coordinates (x, z) of the object point A can be obtained using a known calculation method.

  Here, with reference to FIG. 7, an example of a method for calculating the z coordinate indicating the depth amount using the principle of triangulation will be described. FIG. 7 shows a two-dimensional representation of the observation light orthogonally projected on the XZ plane. Points L and R in the figure indicate the cameras L and R, and points A and B indicate object points. It shall be shown. In this example, a case where the camera L is used as a reference among the left and right cameras L and R will be described.

Now, in FIG. 7, a triangle A having points L, R, and A as vertices or a triangle B having points L, R, and B as vertices will be considered. In the triangle A, each vertex has an angle of point a at point L, b at point R, and point A at 180 ° -ab, and the length of each side is LR = D, LA = r. . Further, in the triangle B, each vertex has an angle of point L at a, point R at b, and point B at 180 ° -ab, and the length of each side is LR = D, LB = r. . Under such conditions, as is apparent from FIG. 7, the angle a of the point L and the angle b of the point R have a relationship of a = 90 ° −θ−x _L and b = 90 ° −θ + x _R , respectively. It will be. Moreover, r can be obtained by r = D × (sinb / sin (180 ° −ab)) using the sine theorem. When the relationship between a, b, and r is obtained, the coordinates (x, z) of the object point are obtained by calculating x = rcosa, z = rsina from x _L and x _R.

  Since the left observation image 401L and the right observation image 401R captured by the imaging elements 171L and 171R of the stereomicroscope 101 have parallax only in the left-right direction (X-axis direction), the y-coordinates should match in principle. However, there may be some deviation due to manufacturing or assembly errors. In that case, after performing the process of correcting the image so that the difference of the y coordinate becomes zero in the previous stage of the process of determining the z coordinate indicating the depth amount, the z coordinate indicating the depth amount is determined. desirable.

  As described above, the depth amount at each corresponding point is determined.

  Returning to the flowchart of FIG. 5, in step S <b> 13, the three-dimensional model generation unit 313 is based on the information obtained from the pair of left and right observation images acquired by the image acquisition unit 311 and the depth amount calculated by the depth amount calculation unit 312. Thus, the depth amount determined at each corresponding point is mapped to the entire plane perpendicular to the central axis of the stereomicroscope 101 to generate a so-called depth map. Then, the three-dimensional model generation unit 313 extracts the surface map at each coordinate on the observation target object 102 extracted from the left observation image 401L and the right observation image 401R with respect to the depth map in which the depth information is expressed. A 3D model corresponding to the observation object 102 is generated by pasting a texture representing color information (so-called texture mapping).

  Although the texture is pasted to the three-dimensional model, it may be generated from only one of the left observation image 401L and the right observation image 401R, but can be obtained from information from one observation image. It is more natural to apply an algorithm that cuts out necessary parts from both observation images in order to deal with occlusion areas that cannot be generated, and it is possible to generate a texture that has fewer surface information missing areas of the observation object 102. .

  In step S14, the three-dimensional observation image generation unit 314 generates a three-dimensional observation image when viewed from a predetermined viewpoint position in the virtual space including the three-dimensional model, based on the generated three-dimensional model, This is supplied to the output unit 315. For example, the three-dimensional observation image generation unit 314 generates a three-dimensional image that is three-dimensionally displayed on the 2D display device 303 as a three-dimensional observation image.

  In step S15, the image output unit 315 outputs the 3D observation image generated by the 3D observation image generation unit 314 to the 2D display device 303 for display. Thereby, as shown in FIG. 3, for example, a triangular prism three-dimensional image or a three-dimensional image 402 displayed on the screen is displayed on the screen of the 2D display device 303.

  As described above, in the control box 301, the two observation images of the left observation image 401L and the right observation image 401R acquired from the stereomicroscope 101 for observing the observation object 102 in an enlarged manner are used, and the observation images are displayed. A depth amount is calculated for each corresponding position, a three-dimensional model of the observation object 102 is generated based on the calculated depth amount, and a three-dimensional observation image of the observation object 102 is generated from the generated three-dimensional model. Generated and displayed on the 2D display device 303.

  As a result, in the conventional confocal microscope, it has been necessary to acquire a plurality of two-dimensional images while moving the observation position in the Z-axis direction in order to obtain information on the height of the observation object. As described above, in the control box 301, predetermined image processing is performed using only two observation images, that is, the left observation image 401L and the right observation image 401R acquired from the stereomicroscope 101 at a time. Thus, a three-dimensional observation image of the observation object can be displayed. As a result, three-dimensional observation of the observation object can be performed easily.

  Further, in the confocal microscope, a driving mechanism in the Z-axis direction is required for moving the observation position in the Z-axis direction. However, in the stereomicroscope 101, in order to capture the left observation image 401L and the right observation image 401R. Since driving in the Z-axis direction is not particularly required, it is not necessary to provide a driving mechanism in the Z-axis direction. Furthermore, since the confocal microscope requires time for driving in the Z-axis direction, for example, when a living medaka is used as an observation object, the observation object moves, and thus a three-dimensional observation image cannot be obtained. . On the other hand, the control box 301 can acquire the left observation image 401L and the right observation image 401R captured by capturing the moment of the observation object by the two imaging elements 171L and 171R. It is possible to construct a three-dimensional observation image of the object.

<2. Second Embodiment>
[Configuration example of microscope system]
Next, a second embodiment of the present invention will be described with reference to FIG. In the second embodiment (FIG. 8) of the present invention, as compared with the first embodiment (FIG. 3), the observation light imaged by the imaging lenses 153L and 153R of the stereomicroscope 101 is photographed. The difference is that deflection elements 154L and 154R are provided to divide the light into eye light and eye light. That is, the observation light imaged by the imaging lens 153L, for example, enters the deflection element 154L configured by a beam splitter prism, passes through the deflection element 154L, and enters the eyepiece lens 155L. The light is reflected by the deflecting element 154L and divided into light for photographing incident on the photographing unit 161L.

  Similarly, the observation light imaged by the imaging lens 153R enters the deflection element 154R having the same configuration as the deflection element 154L, transmits the deflection element 154R, and enters the eyepiece lens 155R. The light is reflected by the deflecting element 154R and divided into photographing light that enters the photographing unit 161R.

  The eyepiece lens 155L and the eyepiece lens 155R form left-eye and right-eye images (hereinafter referred to as observation images) having different parallaxes. The observer can stereoscopically view the image of the observation object 102 by viewing the left-eye and right-eye observation images with the left eye and the right eye via the eyepiece lens 155L and the eyepiece lens 155R, respectively.

  Similarly to the first embodiment, the image sensor 171L of the imaging unit 161L captures an image of the observation light reflected by the deflecting element 154L, and shows the obtained left observation image 401L for the left eye. The signal is supplied to the control box 301 via the cable 162L. Similarly, the imaging element 171R of the imaging unit 161R captures an image of the observation light reflected by the deflecting element 154R, and controls an image signal indicating the obtained right observation image 401R for the right eye via the cable 162R. Supply to box 301.

  The control box 301 has the same functional configuration as the control box 301 in FIG. 4 described above, and performs the same processing as the processing shown in the flowchart in FIG. Using the two observation images of the image 401L and the right observation image 401R, the depth amount is calculated for each corresponding position of the observation images, and the three-dimensional model of the observation object 102 based on the calculated depth amount Is generated. Based on the three-dimensional model, the control box 301 generates an image when the observation object 102 is viewed vertically (hereinafter referred to as a vertically viewed image) and displays the image on the 2D display device 303. For example, a triangular vertical view image displayed on the screen of the 2D display device 303 of FIG. 8 is displayed. Note that the triangular vertical image displayed on the 2D display device 303 in FIG. 8 corresponds to the arrow view of the observation object 102 in the shape reference diagram 400 in FIG.

  As a result, the observer can stereoscopically view the image of the observation object 102 by observing the observation image for the left eye and the right eye through the eyepiece lens 155L and the eyepiece lens 155R, and the 2D display. An image (vertical view image) of the observation object 102 viewed in the vertical direction displayed on the device 303 can also be confirmed.

  In the second embodiment, as in the first embodiment described above, only two observation images, the left observation image 401L and the right observation image 401R, acquired from the stereomicroscope 101 at a time are displayed. Since the vertically-viewed image can be displayed by the predetermined image processing used, three-dimensional observation of the observation target can be performed easily.

<3. Third Embodiment>
[Configuration example of microscope system]
Next, a third embodiment of the present invention will be described with reference to FIG. The third embodiment (FIG. 9) of the present invention is different from the first embodiment (FIG. 3) in that a personal computer 304 is provided instead of the control box 301 of FIG. . Further, the personal computer 304 is connected with a photographing controller 305L that controls the photographing unit 161L and a photographing controller 305R that controls the photographing unit 161R.

  The imaging controller 305L controls the imaging unit 161L to capture an image of the observation light imaged by the imaging lens 153L with the imaging element 171L, and an image signal indicating the obtained left observation image 401L for the left eye is obtained. , Acquired via the cable 162L, and supplied to the personal computer 304. Similarly, the imaging controller 305R controls the imaging unit 161R to capture an image of the observation light imaged by the imaging lens 153R with the imaging element 171R, and shows the obtained right observation image 401R for the right eye. An image signal is acquired via the cable 162R and supplied to the personal computer 304.

  The personal computer 304 has the same functional configuration as the control box 301 of FIG. 4 described above, and performs the same processing as the processing shown in the flowchart of FIG. Using the two observation images of the image 401L and the right observation image 401R, the depth amount is calculated for each corresponding position of the observation images, and the three-dimensional model of the observation object 102 based on the calculated depth amount Is generated. Based on the three-dimensional model, the personal computer 304 generates a three-dimensional observation image of the observation target object 102 or a vertical view image when the observation target object 102 is viewed vertically, and displays the generated image on the 2D display device 303. . For example, the 2D display device 303 in FIG. 9 includes a three-dimensional image of a triangular prism displayed on the screen, a three-dimensional image 402 in FIG. 9, or an arrow view of the observation object 102 in the shape reference diagram 400 in FIG. 9. A vertical view image corresponding to is displayed.

  As a result, the microscope system using the existing stereomicroscope can be easily updated to the microscope system capable of displaying the three-dimensional observation image of the present invention simply by upgrading the program executed by the personal computer 304. it can. Further, since it is not necessary to add a new optical member or the like only by updating the program, a microscope system can be constructed at a lower cost.

  In the third embodiment, as in the first embodiment described above, only two observation images, the left observation image 401L and the right observation image 401R, acquired from the stereomicroscope 101 at a time are displayed. Since the three-dimensional observation image and the vertical vision image can be displayed by the predetermined image processing used, the three-dimensional observation of the observation object can be easily performed.

<4. Fourth Embodiment>
[Configuration example of microscope system]
Next, a fourth embodiment of the present invention will be described with reference to FIG. Compared with the first embodiment (FIG. 3), the fourth embodiment (FIG. 10) of the present invention is provided with a 3D display device 306 instead of the 2D display device 303 of FIG. Is different. In order to simplify the description, the description of the stereomicroscope 101 is omitted and only the image pickup devices 171L and 171R of the photographing units 161L and 161R are described. However, the imaging lenses 153L and 153R are formed by the image pickup devices 171L and 171R. An image formed by the observation light is captured, and image signals indicating the left observation image 401L for the left eye and the right observation image 401R for the right eye are obtained via the cables 162L and 162R, and the control box The points supplied to 301 coincide.

  The control box 301 has the same functional configuration as the control box 301 in FIG. 4 described above, and performs the same processing as the processing shown in the flowchart in FIG. Using the two observation images of the image 401L and the right observation image 401R, the depth amount is calculated for each corresponding position of the observation images, and the three-dimensional model of the observation object 102 based on the calculated depth amount Is generated. The control box 301 (the three-dimensional observation image generation unit 314) renders the three-dimensional model, and two two-dimensional images for the left and right eyes (hereinafter, both the left and right eyes) are used as the three-dimensional observation image. And is supplied to the 3D display device 306.

  The 3D display device 306 is a stereoscopic image display device that displays an image in a stereoscopic manner, and includes, for example, an autostereoscopic display device such as a lenticular method or a parallax barrier method. When the 3D display device 306 displays the left and right binocular image based on the left and right binocular image data supplied from the control box 301, the light emitted from the left eye pixel row is observed by the observer 307. The path of light from each pixel is adjusted so that the light emitted from the right-eye pixel array and the light emitted from the right-eye pixel array enter the right eye of the observer 307.

  As a result, the observer 307 can view the left-eye image with the left eye and the right-eye image with the right eye without using dedicated 3D glasses or the like, and the image of the observation object 102 can be stereoscopically viewed. Can be seen. The 3D display device 306 can be applied not only to the above-described autostereoscopic method, but also to a glasses-type display using polarized light or a liquid crystal shutter.

  Further, in the fourth embodiment, as in the first embodiment described above, only two observation images, the left observation image 401L and the right observation image 401R, which are acquired from the stereomicroscope 101 at a time are displayed. Since the left and right binocular images can be displayed by the predetermined image processing used, three-dimensional observation of the observation object can be performed easily.

<5. Modification>
By converting the data of the three-dimensional model or the three-dimensional observation image generated in the above-described first to fourth embodiments of the present invention into data for three-dimensional CAD (Computer Aided Design), It is also possible to display a three-dimensional model or a three-dimensional observation image generated by the control box 301 or the personal computer 304 on the three-dimensional CAD.

  Specifically, as illustrated in FIG. 4, the CAD data conversion unit 316 includes a three-dimensional model generated by the three-dimensional model generation unit 313 or a three-dimensional observation image generated by the three-dimensional observation image generation unit 314. Is converted into 3D CAD data. The CAD data conversion unit 316 outputs the 3D CAD data obtained by the conversion to a 3D CAD device capable of executing 3D CAD. In the case of the third embodiment, the 3D CAD data may be executed on the personal computer 304 without being output to the 3D CAD apparatus.

  Thus, for example, the design data created by 3D CAD is compared with the observation object 102 (a product actually produced based on the design data) observed by the stereoscopic microscope 101 on the 3D CAD. This makes it possible to intuitively recognize the differences between the two. In addition, when making a three-dimensional data that is difficult to display in a three-dimensional manner, such as a product that is manufactured by an experiment or that is required to have a design, for example, the actual object is observed with the stereomicroscope 101. Then, after generating the data of the three-dimensional observation image by the control box 301 or the like, it is possible to easily make a drawing by converting the data into data for three-dimensional CAD.

  In the above description, the stereomicroscope 101, the photographing unit 161L and the photographing unit 161R, the control box 301, and the 2D display device 303 have been described as separate devices. Alternatively, the image capturing unit 161L and the image capturing unit 161R may be built in the 2D display device 303.

  In the first embodiment, when the observer operates the input device 302 to instruct rotation of the three-dimensional observation image of the observation object 102 displayed in three dimensions in an arbitrary direction, the three-dimensional observation is performed. The image generation unit 314 can generate a three-dimensional observation image when the three-dimensional model is viewed from a viewpoint position corresponding to the rotation in the virtual space of the three-dimensional model, and can display the three-dimensional observation image on the 2D display device 303. For example, the 3D image 402 in FIG. 3 shows a state in which the 3D image of the observation object 102 is rotated on the 2D display device 303. In order to realize such a state with a conventional stereomicroscope, FIG. If the observation object 102 is not tilted or the stereoscopic microscope 101 itself is not tilted, it is impossible to observe at such an angle. Moreover, once such information is acquired, the observation object can be observed from various angles using the information acquired in the past even if the observation object 102 is not actually placed on the stereomicroscope 101 (focus plane). 102 can be observed.

  Thereby, the observer can observe the observation object 102 from arbitrary directions. As the operation method, for example, when “Z key” + “mouse drag” is operated by the input device 302, “pan operation” is selected, and when “X key” + “mouse drag” is operated, If “C key” + “Mouse drag” is operated, “Zoom operation” or the like can be performed. Also, of course, each operation such as “pan operation”, “rotation operation”, and “zoom operation” is displayed as a dedicated toolbar on the 2D display device 303 as an operation GUI, and is selected by the input device 302. Also good.

  Moreover, the illumination method of the stereomicroscope 101 is not limited to the example shown in FIG. 2, and an arbitrary method can be adopted. For example, as shown in FIG. 11, an LED array light source device 501 including an LED array 511L and an LED array 511R on the left and right is attached to the attachment portion of the objective lens 115, and illumination light is emitted from the upper left direction and the upper right direction. May be irradiated to the observation object 102.

  Further, the direction of irradiating the illumination light is not limited to the left and right directions as shown in FIGS. 2 and 11, and for example, the illumination object 102 is irradiated with the illumination light from three or more directions. In this way, the light quantity or spectral distribution of each illumination light may be controlled.

  Further, by connecting the control box 301 and an external recording device (not shown) and storing the left observation image 401L and the three-dimensional model generated from the right eye right observation image 401R, the observation is completed. It is also possible to generate and display a three-dimensional observation image of the observation object 102 from the stored three-dimensional model. Further, at the time of saving, the vertically viewed image when viewed vertically as described in the second embodiment may be stored. Thereby, for example, the observation object 102 can be observed again from an angle for which observation has been forgotten.

  In the fourth embodiment, an example of observing left and right binocular images using an autostereoscopic display device as the 3D display device 306 has been shown. However, the present invention uses left and right eyeglasses and the like. The present invention can also be applied when observing a binocular image.

  In the present specification, the steps for describing a program are not only processes performed in time series in the order described, but also processes that are executed in parallel or individually even if they are not necessarily processed in time series. Is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  Furthermore, the embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

  101 stereo microscope, 102 observation object, 151 objective lens, 152L, 152R afocal zoom system, 153L, 153R imaging lens, 154L, 154R deflecting element, 155L, 155R eyepiece, 161L, 161R imaging unit, 171L, 171R imaging Element 301 control box 302 input device 303 2D display device 304 personal computer 306 3D display device 311 image acquisition unit 312 depth amount calculation unit 313 3D model generation unit 314 3D observation image generation unit 315 Image output unit, 316 CAD data conversion unit

Claims (8)

Image acquisition means for acquiring a pair of left and right observation images imaged by an imaging means arranged on the left and right eye optical paths of a stereomicroscope for observing an observation object in an enlarged manner;
Based on the acquired pair of left and right observation images, calculation means for calculating a depth amount with respect to the central axis of the stereomicroscope at each corresponding position of the pair of left and right observation images;
Three-dimensional model generation means for generating a three-dimensional model corresponding to the observation object based on the depth amount calculated for each corresponding position;
An image processing apparatus comprising: a three-dimensional observation image generation unit configured to generate a three-dimensional observation image that is an image for three-dimensional observation of the observation object based on the generated three-dimensional model. The image processing apparatus according to claim 1, further comprising image output means for outputting the generated three-dimensional observation image to a display device. The display device is a flat image display device that displays an image in a plane,
The three-dimensional observation image generation unit generates a three-dimensional image displayed three-dimensionally on the planar image display device as the three-dimensional observation image,
The image processing apparatus according to claim 2, wherein the image output unit outputs the generated three-dimensional image to the planar image display device. The three-dimensional observation image generation means generates a vertical view image when the observation object is viewed vertically from a direction along the central axis of the stereomicroscope,
The image processing apparatus according to claim 3, wherein the image output unit outputs the generated vertical view image to the planar image display device. The display device is a stereoscopic image display device that stereoscopically displays an image,
The three-dimensional observation image generating means generates a left and right binocular image that is a two-dimensional image for left and right eyes that can be stereoscopically viewed in the stereoscopic image display device as the three-dimensional observation image
The image processing apparatus according to claim 2, wherein the image output unit outputs the generated left and right binocular images to the stereoscopic image display apparatus. The image processing apparatus according to claim 1, further comprising conversion means for converting the data of the three-dimensional model or the three-dimensional observation image into three-dimensional CAD data. The image processing apparatus according to any one of claims 1 to 6, wherein the image processing apparatus is a microscope control apparatus that controls the stereomicroscope. The image processing device
Acquire a pair of left and right observation images imaged by imaging means arranged on the left and right eye optical path of the stereomicroscope that magnifies the observation object,
Based on the acquired pair of left and right observation images, calculate the depth amount with respect to the central axis of the stereomicroscope at each corresponding position of the pair of left and right observation images,
Based on the depth amount calculated for each corresponding position, a three-dimensional model corresponding to the observation object is generated,
An image processing method comprising: generating a three-dimensional observation image, which is an image for three-dimensional observation of the observation object, based on the generated three-dimensional model.

Images