JPH09203854A - Multidirectional image input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a simple device having a wide applicable range and realizing positioning useful on practical use by inputting the image of a standard sample in shape satisfying optical reference in the case of observation or obtaining optical characteristic in the case of observation. SOLUTION: In microscopes 1 and 2, the images of plural minute samples 20 corresponding to different focusing surfaces are picked up while moving the focusing surface by focusing surface control units 7 and 8, and an image signal is inputted in a processor 4. In the processor 4, the image signal is digital- converted by A/D converters 12 and 13 and recorded in memories 14 and 15. The peak coordinate is detected concerning the 3D observed image of the sample 20 inputted from both of microscopes 1 and 2, and defined as a common origin in 3D observation space. The origin of the 3D observed image and information on the scales (x), (y) and (z) obtained in such a way in the image input system of the microscope are utilized for the positioning between the inputted images from the microscopes in an observation mode.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multidirectional image input device configured to input an image of a target object from a plurality of different directions.

[0002]

2. Description of the Related Art Generally, in order to reconstruct a tomographic image or a three-dimensional image (3D image) parallel to an optical axis by using an image input device having an optical image forming system, a plurality of images of an object are obtained. Techniques for inputting from different directions are known. In such a device, it is necessary to align each input image in the tomographic plane.

For example, PJShaw, DAAgard, Y.Hiraoka
and JWSedat, ”Tilted view reconstruction in opti
cal microscopy ”, Biophys. J. 55 (1989) pp. 101-110.
In a microscope equipped with a rotating sample holder, while moving the focus position in the optical axis direction, an image set composed of a plurality of input images is input twice from directions different from each other by 90 °, and these are set to 3 When aligning in dimensional space,
A system has been proposed for calculating the phase correlation between the input image sets of the target object.

[0004]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
In principle, the aperture of the objective lens of a microscope (Numerical Apertu
re: N. A. ) Is sufficiently large, and even if the optical axes are set in two directions orthogonal to each other, it can be used under the condition that the three-dimensional optical transfer characteristics in each direction largely overlap on the spatial frequency.

However, this prior art does not describe the alignment that can be applied when the aperture of the optical imaging system is small and it is difficult to establish correlation between input images, or when it does not depend on the spatial frequency characteristics of the target image. .

Therefore, an object of the present invention is to provide a multi-directional image input device which has a wide application range, is simple, and is practically useful for position alignment.

[0007]

SUMMARY OF THE INVENTION To achieve the above object, the present invention provides a plurality of optical elements arranged to image an object within the range of image input in a space for viewing from different directions. In a multi-directional image input device having a system and a plurality of imaging elements arranged on the image planes of each of the plurality of optical systems, the object is arranged within the range of the image input in which an object exists at the time of image input. A standard sample having a shape satisfying an optical reference setting at the time of observation, a focus position control means for moving a position (focus position) of a focused object surface in an object space of the plurality of optical systems, Corresponds the image information of the standard sample acquired by the image sensor with the in-focus position of the optical system set in the predetermined object space by the in-focus position control means, and the in-focus position information when the image information is acquired. Image of standard sample to be recorded Means and a standard sample observation image space composed of a plurality of standard sample images corresponding to different focus positions recorded in the standard sample recording means while moving the focus position of the optical system by the focus position control means. From the above, there is provided a multi-directional image input device comprising a positional relationship detecting means for detecting the positional relationship between the standard sample observation image spaces corresponding to each of the plurality of optical systems.

Alternatively, a multi-directional image including a plurality of optical systems arranged so as to form an image of an object from different directions and a plurality of image pickup devices arranged on respective image planes of the plurality of optical systems. In the input device, the object is located in the range of the image input at the time of image input, a standard sample having a shape satisfying the optical reference setting at the time of observation, and provided in each of a plurality of optical systems, In a state where the focus position control means for moving the position (focus position) of the focused object surface in the object space, and the focus position of the optical system set in the predetermined object space by the focus position control means, Standard sample image recording means for recording the image of the standard sample acquired by the image pickup device and information of the focus position at the time of capturing this image, and the focus position of the optical system by the focus position control means. While moving the From the standard sample observation image space composed of a plurality of standard sample images corresponding to different focus positions recorded in the quasi-sample image recording means, the standard sample observation image spaces corresponding to each of the plurality of optical systems are Based on the positional relationship between the standard sample observation image space corresponding to each of the plurality of optical systems detected by the positional relationship detecting means and the positional relationship detecting means for detecting the positional relationship, the spatial position of each of the plurality of optical systems Provided is a multidirectional image input device including an optical system position control means for controlling the.

The multidirectional image input device thus constructed satisfies the optical reference setting for observation, or
By inputting an image of a standard sample with a shape that provides optical characteristics during observation, it is possible to estimate an image that directly reflects the spatial characteristics of the optical imaging system. The image information necessary for the registration between the input images at is obtained. INDUSTRIAL APPLICABILITY The present invention can be applied even when the correlation between input images is difficult to take because the aperture of each optical imaging system in the multi-directional image input device is small, and does not depend on the spatial frequency characteristic of the target image. Is wide.

[0010]

Embodiments of the present invention will be described in detail below with reference to the drawings.

FIG. 1 shows an example of the configuration of a multidirectional image input apparatus according to the first embodiment of the present invention. This device is roughly divided into two microscopes 1, a microscope 2 and
The standard sample setting unit 3 and the processor 4 are included.

The microscopes 1 and 2 are constructed in the same manner, and an object space for placing an arbitrary sample (hereinafter referred to as a sample space)
The above optical axes are configured to intersect at a right angle.
The microscopes 1 and 2 have objective lenses 5 and 6 as main components, and focusing surface controllers 7 and 8 for driving and controlling the positions of focused object surfaces (focusing surfaces) in the optical axis direction. , TV cameras 9 and 10 for capturing an image of the sample.

It is assumed that the actual size in the sample space corresponding to the inter-pixel distance of the input image is known by being measured in advance using a standard scale. This is necessary to define a common 3D scale for the observation image space by each of the microscopes 1 and 2. That is, when the z coordinate is defined in the optical axis direction in the observation image space of the microscope 1, the z coordinate is defined in the observation image space of the microscope 1.
The coordinates correspond to the horizontal direction in the image plane.

Therefore, in order to define a common 3D scale, it is necessary to know the actual size in the image and the actual movement distance of the focusing surface.

Further, as a configuration example of the focusing surface controllers 7 and 8, for example, a stepping motor is built in, and stepping motor controllers 17 and 18 in the processor 4 are provided.
The moving distances of the objective lenses 5 and 6 are respectively controlled by the number of pulses generated by, and are set at the focal plane position.

Therefore, the position of the focusing plane corresponding to each of the plurality of input images can be always grasped by the processor 4. An encoder for detecting the positions of the objective lenses 5 and 6 in the optical axis direction may be provided in the focusing surface controllers 7 and 8 and the detected encode signal may be fed back to the processor 4.

With the above configuration, the x, y, z scales of the 3D observation image space formed by arranging a plurality of input images while moving the focusing surface are the x, y, z scales of the microscopes 1 and 2. It is described as a common scale for the input image. In addition, what is necessary for mutual use of images input by the microscopes 1 and 2 is information on the origin common to the microscope image input systems.

In the present embodiment, the origin of the 3D observation image space is determined by the structure using a micro standard sample as described below.

FIG. 2 shows a detailed structural example of the standard sample setting section 3.

In this standard sample setting section 3, the holder 31 for holding the micro sample 20 is a holder rotation control device 32.
The objective lenses 5 and 6 (microscope 1,
The drive control is performed so that the micro sample 20 is installed at a position on the sample space near the intersection of the optical axes of 2).

The operation of the standard sample setting section 3 has a "calibration mode" for obtaining position information and an "observation mode" for observing a target sample. In the calibration mode, the micro sample 20 is placed in the sample space, and the images thereof are simultaneously input by the microscopes 1 and 2.

The constituent operation in the calibration mode related to the present invention will be described below.

In each of the microscopes 1 and 2, the focusing surface controllers 7 and 8 capture images of a plurality of minute samples 20 corresponding to different focusing surfaces while moving the focusing surfaces.
The image signal is input to the processor 4.

In the processor 4, the A / D converter 1
The image signals are digitally converted by 2 and 13 and recorded in the memories 14 and 15, respectively. In this way, a plurality of images with different focusing surfaces are recorded in the memories 14 and 15.

Next, the cumulative adder 16 performs addition between the plurality of images recorded in the memory 14, and the result is recorded in the memory 23 via the memory internal bus 21 as one image. . The peak detector 24 detects two-dimensional peak coordinates from the added image recorded in the memory 23, and the coordinate values are recorded in the memory 25.

The peak of the density in the added image appears corresponding to the position of the standard sample 20 in the 3D space. Then, from each image recorded in the memory 14,
Image density values corresponding to the peak coordinates are read out one after another and recorded in the memory 23 as a one-dimensional profile. Then, the peak detector 24 detects the peak coordinates from the one-dimensional profile recorded in the memory 23, and the coordinate values are recorded in the memory 25.

The above-described operation is similarly performed for the image recorded in the memory 15, and as a result, 3 for the micro sample 20 input from both the microscopes 1 and 2.
For each of the D observation images, the peak coordinates are detected and 3
D is defined as the common origin of the observation space. Note that the memory 2
In 5, the previously obtained scale information in the x, y, z directions is also recorded. The image information of these standard samples and the information of the focus position when capturing (capturing) the image are
Corresponding and stored in the memory.

The origin and the x, y, z scale information of the 3D observation image in each microscope image input system thus obtained are used for alignment between the input images from each microscope in the observation mode. The processor 4 has controllers 17 and 18 for driving and controlling the focusing surface controllers 7 and 8 and a controller 19 for driving and controlling the standard sample setting unit 3. The CPU 22 controls the entire processor 4, and signals are transmitted between the respective components via the internal bus 21.

The minute sample 20 includes a memory 14,
An object smaller than the size in the sample space corresponding to one pixel of the digital image recorded in 15 is used. As an example of the micro sample 20, the cross section may be circular in the direction along the optical axis, and a shape such as a micro metal sphere, a cone, or a cylinder may be used.

When the microscopes 1 and 2 are fluorescence microscopes, microspheres (fluorescent beads) in which a fluorescent dye is permeated may be used.

On the other hand, instead of the micro sample 20, a spherical object which is not so large as the imaging range may be used. Since this spherical object is considered to have a sufficient band in all directions in 3D space, an image enlarged in size similar to the input image when a micro sample is used will be observed. , The peak can be detected. Further, the tip of a sample having a sharp tip as shown in FIG. 3 may be used as a pseudo micro sample. As an example of a sample having a sharp tip, a micro needle or the like formed by a semiconductor process may be used.

Next, the operation of the multi-directional image input device thus constructed will be described.

This multi-directional image input device is used for a micro sample 3D.
By inputting an image and detecting the peak coordinates, it has the effect of setting the origin of the observed image space coordinates.

FIG. 4 shows the concept of the sample space and the 3D observation image space.

Generally, an optical image forming system is subject to a strong band limitation in the optical axis direction, but in the case of an optical image forming system having a large aperture (NA) such as used in an optical microscope, the depth of focus is large. Since the 3D image of the micro sample is relatively shallow, the density peak in the added image appears at the position where the micro sample exists although it extends longer in the optical axis direction than in the image plane direction.

Therefore, the origin of the coordinates in the observed image space can be set by detecting the peak of such a 3D image. In the multi-optical axis microscope as shown in the present embodiment, a micro sample installed in the sample space is observed from each microscope, and if the origin of each observation space is set, the images input by each microscope have a common image. It can be handled in 3D space.

With reference to FIG. 5, a method for detecting the peak coordinates from the 3D image of the micro sample in the present embodiment will be described.

First, in the XY plane parallel to the image plane,
In order to detect the peak, a plurality of images input while moving the focusing surface are cumulatively added in the Z direction, and 2D search is performed within the cumulatively added image to detect the peak position. With this, the origin for the X and Y coordinates can be detected. Then XY
The peak coordinates are detected from the one-dimensional profile in the Z direction created by reading out the pixel density value corresponding to the peak coordinates of the surface from each input image. In this way, the origin in the Z direction can be detected.

As described above, according to the first embodiment, it is possible to input a 3D image directly reflecting the 3D spatial characteristics of the optical imaging system by using a micro sample or an object corresponding thereto as a standard sample. Therefore, the origin can be easily set by detecting the peak of the 3D observation image.

By performing such an operation for each of a plurality of image input systems with respect to the same standard sample image, each 3D image can be obtained.
The origin obtained for the observation image space can be treated as a common origin, and registration between input images from different directions can be realized. Further, since the inter-pixel distance and the position of the focusing surface in each input image are known in the processor 4, these and the position information of the origin define a common 3D coordinate for each 3D observation image space. .

Furthermore, according to the peak coordinate detection method shown in the present embodiment, peak detection may be carried out once for each of 2D and 1D, which is simpler than a direct search for the peak coordinates from the 3D observation space. Become. That is, it is not necessary to directly search the 3D image including a large number of pixels, and the peak coordinates are detected by a simple method.

Next, FIG. 6 shows a configuration example of a multi-directional image input device as a second embodiment according to the present invention, which will be described. 6 which are the same as those of the first embodiment shown in FIG. 1 are designated by the same reference numerals and the description thereof will be omitted.

In this embodiment, the microscope 1 is constructed so as to be held by the xyz automatic stage 34. A controller 35 is provided on the internal bus 21 in the processor 4, and a control signal from the controller 35 drives the xyz automatic stage 34 to finely adjust the spatial position of the microscope 1.

The operation of the second embodiment thus constructed will be described.

In this embodiment, the origin of the 3D observation image space detected by the same configuration operation as that of the first embodiment is 3
It has the function of finely adjusting the position of the microscope 1 so that it is located at the center of the D observation image space.

First, similarly to the first embodiment, when the origin on the 3D observation image space in the microscope 1 is observed, the distance between the center of the 3D observation image space and the origin in each of the x, y, and z directions. Is calculated, and x− is calculated according to the distance information.
The yz automatic stage 34 is drive-controlled. Then, the standard sample image is input again, and the origin of the 3D observation image space is detected.

By repeating the above operation several times, 3
The origin and the center become coincident with each other in the D observation image space.

According to the present embodiment, the condition that the origin coincides with the center in the 3D observation image space is obtained, so that the largest view angle size can be used when the input images from each microscope are mutually used in the observation mode. . That is, it is not necessary to shift the address of the input image when aligning the input images from each microscope, and the reconstructed image does not become smaller than the input image.

In this embodiment, the microscope 2 may be provided with an xyz automatic stage so that both microscopes are similarly adjusted in position.

Next, FIG. 7 shows an example of the configuration of a multidirectional image input apparatus as a third embodiment according to the present invention.

This embodiment is similar to the first embodiment in that
The present invention relates to a method of estimating an origin from a parallel projection image when obtaining the origin in a 3D observation image space from an input image of a minute sample. FIG. 7 shows the configuration of the processor 40 of the present embodiment, but other configurations of the microscope and the like are similar to those of the first embodiment.

In this multidirectional image input device, as in the first embodiment, the image of the micro sample input while the focusing planes are moved by the microscopes 1 and 2 is A /
It is converted into a digital image signal by the D converters 41 and 42 and recorded in the memories 43 and 44. 3D composed of a plurality of images with different focal planes recorded in the memory 43
The observation image is rotated by the image rotation unit 45 by a predetermined angle φ. Note that the image rotation unit 45 stores the read address and the interpolation coefficient for the rotation operation of the angle φ.
M and a product-sum calculation device are included, and the 3D observation image is configured to be rotated while being interpolated.

3D rotated by a predetermined angle φ
The observed image is recorded in the memory 46, and then the cumulative adder 47
Is cumulatively added in the Z-direction, and the angle φ is stored in the memory 48.
The memory 4 is used as a two-dimensional (2D) projected image for the direction.
8 is recorded. The peak position is detected by the peak detector 49 from the 2D projection image recorded in the memory 48. Such an operation is repeated for different rotation angles φ, and the peak coordinates in each angular direction are detected.

Next, the detected peak coordinates are sent to the sine curve regression analyzer 50, and as a result of the predetermined analysis, the origin coordinates are detected and recorded in the memory 51. The above operation is similarly performed on the 3D observation image of the microscope 2 recorded in the memory 44.

Further, in the processor 40, controllers 52, 53, 54 corresponding to the controllers 17, 18, 19 of the first embodiment are provided. Also processor 4
The overall control of 0 is performed by the CPU 55, and the signal transmission between the respective components is performed via the internal bus 56.

Next, the operation of the multidirectional image input device thus constructed will be described with reference to the conceptual diagram of FIG.
In this embodiment, the origin is detected from the 3D observation image of the micro sample by the following actions.

First, a projection image 1 shown in FIG. 8 is created by projecting a 3D observation image in the optical axis direction in parallel, and the origin in the XY plane is detected.

Next, it is considered that the 3D observation image is projected onto the optical axis in the direction of the angle φ with the center of the 3D observation space as the center to create the projection image 2 shown in FIG. Although the image is described as being projected in the angular direction φ in FIG. 8,
The configuration shown in FIG. 7 is realized by rotating the image itself by the angle −φ and then projecting it in the Z direction.

However, it is clear that the same projection image can be obtained with both. The position where the micro sample actually exists is 3
Assuming that the distance D is displaced from the center of the D observation space in the optical axis direction, the displacement of the peak coordinates in the projected image from the center is expressed by the following equation.

[0060]

[Equation 1]

Therefore, it is possible to estimate d by detecting peak coordinates for several kinds of angular directions φ and performing regression approximation on them with a sine curve. It should be noted that the N.V. A. Since the angular direction φ in which the peak can be detected is limited to the range in which the following expression is satisfied, several angular directions φ may be selected within the range.

[0062]

[Equation 2]

According to this embodiment, since the peak coordinates are detected from the 2D projected image when the origin is detected, the peak detection can be performed more easily than the peak coordinates are directly detected from the 3D observation image. Further, since the origin is estimated from the peak values detected from a plurality of projected images, the detection error can be canceled and the most probable origin position can be estimated.

The present embodiment may be operated as follows with the same configuration as in FIG. First, the origin in the XY plane is detected from the 2D projection image in the direction parallel to the optical axis. Next, an image that passes through the XY origin detected from the 3D observation image recorded in the memory 503 or 504 and is parallel to the rotation surface is read. The read 2D image is rotated by a predetermined angle φ and then cumulatively added in the Z direction to form a 1D parallel projection image. Peak detection is performed on this 1D profile. With the above configuration, the peak detection is a search on 1D, so that it can be performed more easily.
Moreover, the origin can be detected accurately.

Next, FIG. 9 shows an example of the configuration of a multi-directional image input device as a fourth embodiment according to the present invention.

This multi-directional image input device is a device for obtaining the relative positional relationship between the respective 3D observation image spaces by utilizing the correlation between the input images from each direction with respect to the micro sample in the multi-optical axis image input system. is there.

The processor 600 of this embodiment shown in FIG.
Other configurations other than the above are the same as those of the first embodiment shown in FIG. That is, the image of the micro sample input while being moved on the focal plane by the microscopes 1 and 2 is converted into a digital image signal by the A / D converters 61 and 62 in the processor 60 and recorded in the memories 63 and 64. It

Next, in the DSP 65 configured to execute the FFT, a 3D Fourier transform is applied to each of the memories 63 and 64 with respect to the 3D observation image composed of the recorded images having different focal planes. Done, memory 6 again
3, 64. Subsequently, in the product-sum calculator 66, a predetermined product-sum calculation is performed using the 3D Fourier images recorded in the memories 63 and 64, and the result is
It is recorded in the memory 64. 3 re-recorded in memory 64
The DSP 65 performs 3D inverse Fourier transform on the D Fourier image, and the result is recorded in the memory 63.

The 3D correlation between the 3D observation images is calculated by the above-mentioned configuration operation. The peak detector 67 detects peak coordinates from the 3D correlation image recorded in the memory 63, and the coordinate values are recorded in the memory 68. The 3D correlation peak coordinates recorded in the memory 68 are used in the observation mode as a relative positional relationship between the observation images by each input optical system.
Further, in the processor 60, controllers 69, 7 corresponding to the controllers 17, 18, 19 of the first embodiment are provided.
0, 71 are provided. The entire processor 60 is controlled by the CPU 72, and signals are transmitted between the respective components via the internal bus 73.

The operation of the fourth embodiment thus constructed will be described. This embodiment is 3 defined by the following equation.
It has the function of calculating the correlation between D observation images.

[0071]

(Equation 3)

However, g1 (r) and g2 (r) are the first and second 3D observation images in the multi-optical axis image input system, respectively. Both vectors r and t are coordinate vectors defined in the 3D orthogonal coordinate space. When using a minute sample, the equation (3) can be developed as follows.

[0073]

(Equation 4)

However, h1 (r) and h2 (r) are respectively the first and second 3D response functions in the multi-optical axis image input system,
δ (r) is a delta function representing a small sample, rs is the first and second
3D vector representing relative displacement between 3D observation images of
And * are 3D convolution operators. When the equation (4) is calculated, a peak appears at the position of the point rs in the 3D correlation image s (t), and the relative positional relationship can be known by detecting it. In this embodiment, 3D
When obtaining the correlation, the calculation using the 3D Fourier transform is performed as shown below. Expression (3) can be rewritten as the following expression.

[0075]

(Equation 5)

However, F _3D {} is an operator of 3D Fourier transform, F ^{* 3} _D {} is an operator of 3D Fourier transform and complex co-contract, and F _3D ^-1 {} is an operator of 3D inverse Fourier transform. 3D observation images g1 (r) and g2 (r) are 3D
The Fourier transform images G1 (u) and G2 (u) (where u is the spatial frequency coordinate of the 3D orthogonal space) are rewritten as follows.

[0077]

(Equation 6)

In this case, the calculation performed in the 3D Fourier space is expressed by the following equation.

[0079]

(Equation 7)

The product-sum calculator 66 executes the calculation based on the equation (7) on the two 3D Fourier images. From the correlation image of the two 3D observation images calculated in this way,
The relative positional relationship can be obtained by detecting the peak coordinates.

By the way, N.V. of each optical system of the multi-optical axis optical image forming system. A. Unless 3 is very large, the correlation between the 3D observation images is weak, and it is difficult to detect the correlation peak with high accuracy for any sample.

However, according to this embodiment, since a minute sample that can be treated as an ideal point object is used, a 3D observation image representing the original image forming characteristics of the optical image forming system can be obtained, and the most ideal 3D correlation can be obtained. You can ask for a statue. That is, if there is even a slight correlation between the optical imaging systems, it becomes possible to detect the correlation peak using the 3D observation image of the minute sample.

Therefore, according to the correlation method, there is no need for the operation for obtaining the coordinates common to each 3D observation space as shown in the first to third embodiments, and the relative positional relationship between the 3D observation images is Since it is directly required, less information needs to be recorded, and at the same time, the alignment process becomes simple.

Next, FIG. 10 shows an example of the configuration of a multidirectional image input apparatus as a fifth embodiment according to the present invention.

This multi-directional image input device irradiates a fluorescent dye solution with laser light focused by an objective lens,
It is configured so as to bring about the same effect as placing a micro sample in the sample space.

In this multidirectional image input device, the microscopes 81 and 82 have objective lenses 83 and 84, respectively, and are arranged so that their optical axes are orthogonal to each other in the sample space, as in the first embodiment. , 82 are similarly provided with a focusing surface controller and a TV camera, but the illustration thereof is omitted here.

A container 85 is installed in the sample space in the calibration mode, and a fluorescent dye solution 86 is injected therein. Further, the laser optical system is configured as follows.

The laser beam emitted from the laser 87 is converted into a desired beam size by the laser beam expander 88 and introduced into the microscope 81 by the mirror 89. A mirror 90 is installed in the microscope 81, and the laser light is converged in the container 85 by using the image forming optical system of the objective lens 83. The microscope 81 is provided with a focusing surface controller (not shown), but when the laser light is guided into the microscope 81, the focusing surface controller causes the focusing surface of the microscope 81 to move in the 3D observation image space. It is set at the center of the defined coordinate in the optical axis direction. At this time, the microscope 82
Then, a plurality of images are input while moving the focusing surface and transferred to the processor 80. The processor 80 is configured similarly to the processor 4 of the first embodiment, and the peak coordinates are detected from the 3D observation image input using the microscope 82.

In the observation mode, instead of the container 85, another container having the same specifications as the container 85 in which the sample is injected together with the solution is installed and the internal sample is observed. The objective lenses 83 and 84 may be water immersion lenses, and a new container may be provided so that both the objective lenses and the container 85 are immersed in the liquid. In that case, the container 85 is made of a substance having a refractive index close to that of the surrounding liquid.

Next, the operation of the present embodiment will be described.

In the present embodiment, the fluorescent dye solution 86 is excited by the laser light spot created in the container 85 by utilizing the optical imaging system of the microscope 81, so that the micro sample is placed in the sample space. Bring about the action of. Since laser light has excellent spatial coherence, ideal spot light can be created. Since the laser light spot appears at a position corresponding to the origin on the 3D observation space of the microscope 81, by detecting the position from the 3D observation image of the microscope 82, the relative positional relationship between both optical imaging systems can be known. it can.

Therefore, according to this embodiment, it is possible to obtain the same effect as that of placing a micro sample in the sample space by the simple method of placing the container 85 in the sample space, and therefore, the apparatus configuration and operation can be performed. It becomes easy.

Next explained is a multi-directional image input device according to the sixth embodiment.

In the sixth embodiment, the fluorescent dye solution is irradiated with the laser light focused by the objective lens. However, since the two-photon excitation phenomenon is used, strong pulsed light in the infrared region is generated. The model is used. During the continuous irradiation of the laser pulsed light, the microscope 82 releases the shutter of the TV camera until an image with sufficient contrast is obtained, and then the image signal is transferred to the processor 80.

The operation of the sixth embodiment will be described.

The present embodiment utilizes the two-photon excitation phenomenon of the fluorescent dye by using pulsed laser light having a wavelength twice the excitation wavelength of the fluorescent substance as the laser light source.
Fluorescence due to two-photon excitation is usually extremely difficult to observe, but observable fluorescence is generated in a high-energy local space created by converging strong laser pulse light. Since such two-photon excitation fluorescence is generated only in an extremely minute space, an ideal minute point light source that emits light in a range far narrower than the depth of focus of the optical imaging system can be obtained.

Therefore, according to the present embodiment, it is possible to observe an ideal minute fluorescent point, so that it is possible to perform the alignment with excellent accuracy.

Next, FIG. 11 shows an example of the configuration of a multidirectional image input apparatus according to the seventh embodiment of the present invention.

This multidirectional image input device is configured to use a solution in which a sample to be observed and a micro sample used for alignment are mixed.

This multi-directional image input device includes a microscope 91,
92 is arranged so that the optical axes thereof are orthogonal to each other in the sample space, as in the first embodiment described above. The microscopes 91, 9
Similar to the first embodiment, 2 is provided with an objective lens, a focusing surface controller, and a TV camera, but their illustration is omitted.

First, a container 93 is installed in the sample space,
A sample 95 to be observed is injected together with the micro fluorescent spheres 94 at an appropriate density. It should be noted that the micro fluorescent sphere 94 and the sample 95 have fluorescence spectra in different wavelength regions.
It is dyed with each type of fluorescent dye. Further, two types of color filters 96 and 97 having different transmission wavelength regions are installed on the optical path of the microscope 91, and one of them is used according to need. Switching between these color filters 96 and 97 is performed by the filter changing unit 98.

In such a configuration, in the calibration mode, the color filter 96 that selectively transmits the fluorescent wavelength of the micro fluorescent sphere 94 is installed on the optical path, and the image of the micro fluorescent sphere 94 is input by the microscope 91. It In the observation mode, the color filter 97 that selectively transmits the fluorescence wavelength of the sample 95 is installed on the optical path, and the sample 95 is observed.

Although not shown in FIG. 11, the structure as described above is also provided in the microscope 92 as well.
The image of the micro fluorescent spheres 94 input by each of the microscopes 91 and 92 is input to the processor 99, and the peak coordinates are detected by performing the same processing as in the first embodiment.
However, in the present embodiment, a 3D observation image space is defined in which peak coordinates are detected for each of the images of the plurality of micro fluorescent spheres 94 captured in the image, and the position coordinates thereof can all be recognized at a common address. To be done. A controller 101 for controlling the filter changing unit 98 on the internal bus 100 is provided in the processor 99, but other configurations are the same as the processor 99 of the first embodiment.
The configuration is the same as

Next, the operation of this embodiment will be described.

In this embodiment, the image of the micro fluorescent spheres 94 is selected in the calibration mode by changing the color filters 96 and 97 without changing the configuration in the sample space between the calibration mode and the observation mode. It is configured to be input manually.

Therefore, according to this embodiment, the structure and operation in the sample space can be simplified. Further, since the time difference between the calibration mode and the observation mode can be shortened, more accurate alignment can be realized. Further, since the alignment of the plurality of micro fluorescent balls 94 can be used, there is an effect that the accuracy can be improved.

Although the embodiments have been described above, the present invention also includes the following inventions.

(1) A plurality of optical systems arranged so as to form an image of an object within an image input range in a space for observing from different directions, and arranged on each image plane of the plurality of optical systems. In a multi-directional image input device having a plurality of image pickup elements, a standard sample having a shape that can be placed in the range of the image input where an object exists at the time of image input and can be set as an optical reference during observation. A focusing position control means for moving a position (focusing position) of a focused object surface in the object space of the plurality of optical systems; and a focusing position of the optical system by the focusing position control means. Standard sample image recording means for correspondingly recording image information of the standard sample acquired by the image pickup device in a state of being set in the object space and information of a focus position at the time of acquiring the image information; Optical by position control means From the standard sample observation image space composed of a plurality of images of standard samples corresponding to different focus positions recorded in the standard sample recording means while moving the focus position of And a positional relationship detecting means for detecting the positional relationship between the standard sample observation image spaces.

The embodiment relating to the present invention is as follows.
Three or four embodiments correspond.

According to the present invention, it is possible to estimate an image that directly reflects the spatial characteristic of the optical image forming system by inputting the image of the standard sample having the shape capable of obtaining the optical characteristic during observation. Therefore, it is possible to obtain the image information necessary for the alignment between the input images in the multidirectional image input device. Therefore, the present invention can be applied to the case where the correlation between the input images is difficult to be taken because the aperture of each optical imaging system in the multidirectional image input device is small, and does not depend on the spatial frequency characteristic of the target image. It is possible to provide a positioning method which has a wide range and is simple and practically useful.

(2) In the multidirectional image input device described in the item (1), the positional relationship detecting means is a standard sample observed image space for a standard sample observation image space corresponding to each of a plurality of optical systems. The feature is that the positional relationship between the standard sample observation image spaces is detected by the absolute address coordinates with the position of the image of the origin as the origin.

(3) In the multi-directional image input device described in the item (2), the positional relationship detecting means can detect the standard sample from the standard sample observation image space corresponding to each of the plurality of optical systems. It is characterized by having a function of detecting the most focused position as a peak position.

(4) In the multi-directional image input device described in the item (3), the positional relationship detecting means is different for each of the plurality of optical systems and different from each other recorded in the standard sample image recording means. Image adding means for adding the image information of a plurality of standard samples corresponding to the focal position, first peak position detecting means for detecting the peak position of the pixel density value from the added image information, and the first peak position detecting means. A second peak position detecting unit for detecting a peak position from pixel density values of images of a plurality of standard samples corresponding to different in-focus positions at the peak position detected by the peak position detecting unit; The relative positional relationship between the arrangement position of the optical system and the peak position corresponding to each optical system obtained by the second peak detecting means is obtained,
The positions of the standard sample observation image spaces corresponding to each of the plurality of optical systems are determined by determining that the peak positions for each optical system obtained by the first and second peak position detecting means are at the same position. A multidirectional image input device characterized by obtaining a relationship.

(5) In the multidirectional image input device described in the item (4), the first peak position detecting means and the second peak position detecting means in the positional relationship detecting means are configured and used in the same manner. Or one of the peak position detecting means is switched and used.

(6) In the multi-directional image input device described in the item (3), the positional relationship detecting means includes a plurality of standard samples corresponding to different focus positions recorded in the standard sample image information recording means. Parallel projection means for generating a parallel projection image in a predetermined angle direction with respect to the optical axis direction by using the image of
It is characterized by comprising peak position detecting means for detecting the peak position from the parallel projection image, and position estimating means for estimating the position of the standard sample using the peak position.

The embodiment relating to the present invention corresponds to the first embodiment.

According to the present invention, a plurality of optical systems are detected by detecting image information (eg, peak coordinates) which is an index effective for alignment between input images in the multidirectional image input device from the image of the standard sample. It is possible to easily set the coordinates of the standard sample observation image space corresponding to each of the above. And
By performing such an operation with respect to the same standard sample image in each of a plurality of image input systems, it becomes possible to treat the coordinates obtained for each standard sample observation image space as common coordinates, and from different directions. Registration between the input images is realized.

(7) In the multidirectional image input device described in the item (1), the positional relationship detecting means detects the correlation between the standard sample observation image spaces with respect to each of the plurality of optical systems to obtain a standard image. The feature is that the positional relationship between the sample observation image spaces is detected.

(8) In the multidirectional image input device described in the item (7), the positional relationship detecting means performs a Fourier transform on the standard sample observed image space for each of the plurality of optical systems. And a cross-correlation calculation means for performing a cross-correlation calculation defined in the Fourier space from the standard sample observed image space for each of the Fourier-transformed optical systems, and a cross-correlation in the Fourier space calculated by the cross-correlation calculation means. An inverse Fourier transform means for performing an inverse Fourier transform on the image, and a peak position detecting means for detecting a peak position from the cross-correlation image subjected to the inverse Fourier transform are provided.

(9) In the multidirectional image input device described in the item (8), the Fourier transform means and the inverse Fourier transform means in the positional relationship detecting means are constituted by the same means. .

The fourth embodiment corresponds to the embodiment relating to the present invention.

According to the present invention, by inputting the image of the standard sample whose shape is known in advance, it is possible to estimate the image that directly reflects the spatial characteristic of the optical image forming system, which is the most ideal. The correlation image between the standard sample observation image spaces can be obtained. According to the correlation method, since the relative positional relationship between the standard sample observation images is directly obtained,
The amount of information to be recorded is small, and at the same time, the alignment process becomes simple.

(10) In the multidirectional image input device according to item (1), the standard sample has a diameter smaller than the size of the object space corresponding to one pixel of the image recorded in the standard sample image recording means. It is characterized by being composed of objects.

The first embodiment corresponds to the embodiment of the present invention.

According to the present invention, one of the input images is input to the standard sample.
By using a minute object smaller than the size corresponding to a pixel, an image that directly reflects the spatial characteristics of the optical imaging system can be input. Therefore, it is possible to obtain the most ideal input image for performing the processing according to the present invention.

(11) In the multi-directional image input device described in the item (1), the standard sample has a cross-sectional shape capable of obtaining characteristics of an optical system of the multi-directional image input device during observation. Characterize.

(12) In the multidirectional image input device described in the item (1), the standard sample has a circular cross section in a plane including two optical axes or in a plane including each optical axis. And is a spherical, conical, or cylindrical object. The embodiment relating to the present invention corresponds to the first embodiment.

According to the present invention, a spherical object is considered to have a sufficient band for all directions in 3D space.
An image that is similar in size to the input image when a micro sample is used and whose size is enlarged is observed, and the action and effect as a standard sample can be obtained. In addition, since the sample does not necessarily have to be minute, it is easy to handle and practically advantageous.

(13) In the multidirectional image input apparatus described in the item (1), the standard sample is the tip of a sharp object, and the diameter of the object is the diameter of the image recorded in the standard sample image recording means. It is characterized in that the range is smaller than the size of the object space corresponding to one pixel.

The embodiment relating to the present invention corresponds to the first embodiment.

According to the present invention, the tip of a sharp object is used in order to use the most ideal minute object as a standard sample, and it is not necessary to make the standard sample itself minute, It is possible to obtain the same operational effect as when an object is used. Therefore, handling is simple and the most ideal standard sample image can be input.

(14) A plurality of optical systems arranged so as to form an image of an object within the range of image input in the space for observing from different directions, and arranged on the image plane of each of the plurality of optical systems. In a multi-directional image input device having a plurality of image pickup elements, a standard sample having a shape that can be placed in the range of the image input where an object exists at the time of image input and can be set as an optical reference during observation. , A focus position control means provided in each of the plurality of optical systems for moving a position (focus position) of a focused object surface in the object space, and a focus position control means for adjusting the focus of the optical system. Standard sample image recording means for recording the image information of the standard sample acquired by the image pickup device in a state where the focal position is set in a predetermined object space and the information of the focus position when the image information is acquired in association with each other. And the in-focus position While moving the focus position of the optical system by the control means, from the standard sample observation image space composed of the images of the plurality of standard samples corresponding to different focus positions recorded in the standard sample image recording means, This standard sample observation image space corresponding to each of the optical systems, and a positional relationship detecting means for detecting the positional relationship between the standard sample observation image space and the standard sample observed image space corresponding to each of the plurality of optical systems detected by the positional relationship detecting means. An optical system position control means for controlling the spatial position of each of the plurality of optical systems based on the positional relationship between them, and a multidirectional image input device.

The second embodiment corresponds to the embodiment relating to the present invention.

According to the present invention, the origin is aligned with the center in the standard sample observed image space based on the positional relationship between the standard sample observed image spaces corresponding to each of the plurality of optical systems detected by the positional relationship detecting means. It is configured to finely adjust the spatial position of each of the plurality of optical systems.

Therefore, according to the present invention, the largest view angle size can be used when the input images from the respective microscopes are mutually used. That is, it is not necessary to shift the address of the input image when aligning the input images from each microscope, and the reconstructed image does not become smaller than the input image.

(15) Multi-direction comprising a plurality of optical systems arranged so as to form an image of an object from different directions and a plurality of image pickup devices arranged on respective image planes of the plurality of optical systems In the image input device, a container containing a fluorescent substance, which is arranged at a substantially position where an object is present at the time of image input, a light source means for generating light for exciting the fluorescent substance in the container, and a plurality of optical systems Optical means for guiding the excitation light to the first optical system so that the excitation light emitted by the light source means is converged in the container by using the first optical system which is one of
Focusing position control means provided in each of the plurality of optical systems for moving the position (focusing position) of the focused object surface in the object space, and the focusing position of the optical system is determined by the focusing position control means. A fluorescent image recording means for recording the fluorescent image acquired by the image sensor in the state of being set in the object space and the information of the focus position at the time of capturing the fluorescent image in association with each other,
In the second optical system, which is one of the plurality of optical systems and is different from the first optical system, it is recorded in the fluorescent image recording means while moving the focus position of the optical system by the focus position control means. And a positional relationship detecting means for detecting the positional relationship between the first and second fluorescence observation image spaces from the fluorescence observation image space formed of a plurality of fluorescence images corresponding to different in-focus positions. Characterize.

The fifth embodiment corresponds to the embodiment of the present invention.

According to the present invention, a micro sample is placed in the sample space by irradiating the fluorescent material placed in the sample space with the converged laser light utilizing each optical imaging system in the multidirectional image input device. It is configured to bring about the same effect as. Therefore, according to the present invention, it is possible to obtain the same effect as that of placing a micro sample in the sample space by a simple method of placing the container into which the fluorescent substance is injected in the sample space, and thus the device configuration and operation are simple. become.

(16) In the multidirectional image input device described in the item (3), the light source means is a laser light source means for generating pulsed laser light having a wavelength twice the excitation wavelength of the fluorescent substance in the container. It is characterized by

The sixth embodiment corresponds to the embodiment of the present invention.

According to the present invention, when irradiating the fluorescent material placed in the sample space with the laser light converged by using each optical imaging system in the multidirectional image input device, the laser light source is controlled to have the excitation wavelength of the fluorescent material. By using a pulsed laser beam having a double wavelength, a two-photon excitation phenomenon is caused and an extremely small fluorescent spot is realized. Therefore, according to the present invention, it is possible to obtain the same effect as that of placing a micro sample in the sample space by a simple method of placing the container into which the fluorescent substance is injected in the sample space, and thus the device configuration and operation are simple. In addition, since it is possible to observe an ideal minute fluorescent point, it is possible to perform highly accurate alignment.

(17) Multi-direction comprising a plurality of optical systems arranged so as to form an image of an object from different directions, and a plurality of image pickup devices arranged on respective image planes of the plurality of optical systems In the image input device, a container into which a standard sample that emits fluorescence having a wavelength different from the fluorescence wavelength of the target is injected together with a target that emits fluorescence, a first color filter that transmits the fluorescence wavelength of the target, and A second color filter that transmits the fluorescence wavelength of the standard sample; and a color filter changing unit that is provided in each of the plurality of optical systems and that installs one of the first and second color filters on the optical path of the optical system. And a second color filter, which is provided in each of the plurality of optical systems and which is provided with an in-focus position control means for moving a position (in-focus position) of an in-focus object surface in the object space, and a second color filter. By focusing position control means Standard sample image recording in which the image of the standard sample acquired by the image sensor and the information of the focus position at the time of capturing this image are recorded in association with each other while the focus position of the optical system is set in a predetermined object space. Means and a standard sample observation image composed of a plurality of standard sample images corresponding to different focus positions recorded in the standard sample image recording means while moving the focus position of the optical system by the focus position control means. And a positional relationship detecting means for detecting a positional relationship between the standard sample observation image spaces corresponding to each of the plurality of optical systems from the space.

The seventh embodiment corresponds to the embodiment of the present invention.

The present invention uses a solution in which a fluorescent sample to be observed and a standard sample which emits fluorescence having a wavelength different from the fluorescent wavelength of the object used for alignment are mixed, and the fluorescent wavelength of the standard sample is selectively changed. It is configured such that only a standard sample image can be input using a color filter that transmits light. Therefore, according to the present invention, the configuration and operation in the sample space can be simplified. Further, since the time difference between the work of inputting the standard sample image for the alignment and the observation work of the target object can be shortened, more accurate alignment can be realized.

[0145]

As described in detail above, according to the present invention, the present invention can be applied to the case where the correlation between the input images is difficult because the aperture of each optical imaging system is small, and the spatial frequency characteristic of the target image can be obtained. It is possible to provide a multi-directional image input device which has a wide application range, is not dependent on the above, is simple, and is capable of practically useful positioning.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration example of a multi-directional image input device as a first embodiment according to the present invention.

FIG. 2 is a diagram showing a detailed configuration example of a standard sample installation unit shown in FIG.

FIG. 3 is a diagram showing the appearance of a pseudo minute sample in which the tip of a sample having a sharp tip used in the present embodiment is simulated.

FIG. 4 is a diagram showing a concept of a sample space and a 3D observation image space in the present embodiment.

FIG. 5 is a diagram for explaining a method for detecting peak coordinates from a 3D image of a micro sample in the present embodiment.

FIG. 6 is a diagram showing a configuration example of a multi-directional image input device according to a second embodiment of the present invention.

FIG. 7 is a diagram showing a configuration example of a multi-directional image input device according to a third embodiment of the present invention.

FIG. 8 is a diagram for explaining the operation of the multi-directional image input device according to the third embodiment.

FIG. 9 is a diagram showing a configuration example of a multi-directional image input device according to a fourth embodiment of the present invention.

FIG. 10 is a diagram showing a configuration example of a multi-directional image input device as a fifth embodiment according to the present invention.

FIG. 11 is a diagram showing a configuration example of a multi-directional image input device as a seventh embodiment according to the present invention.

[Explanation of symbols]

 1, 2 ... Microscope 3 ... Standard sample setting unit 4 ... Processor 5, 6 ... Objective lens 7, 8 ... Focusing plane controller 9, 10 ... TV camera 12, 13 ... A / D converter 14, 15, 23, 25 ... Memory 16 ... Cumulative adder 17, 18, 19 ... Controller 20 ... Micro sample 21 ... Internal bus 22 ... CPU 24 ... Peak detector

Claims (3)

[Claims] 1. A plurality of optical systems arranged to form an image of an object within a range of image input in a space for observing from different directions, and arranged on each image plane of the plurality of optical systems. In a multi-directional image input device having a plurality of imaging elements, a standard sample having a shape that can be set in the range of the image input where an object is present at the time of image input, and can be set as an optical reference when observing, Focusing position control means for moving the position (focusing position) of the focused object surface in the object space of the plurality of optical systems, and the focusing position control means for adjusting the focusing position of the optical system to a predetermined object. Standard sample image recording means for recording image information of the standard sample acquired by the image pickup device in a state of being set in space and information of a focus position at the time of acquiring the image information, and the focus position. Of the optical system by the control means From the standard sample observation image space composed of images of a plurality of standard samples corresponding to different focus positions recorded in the standard sample recording means while moving the focus position, each of the plurality of optical systems corresponding to A multidirectional image input device, comprising: a positional relationship detecting means for detecting a positional relationship between standard sample observation image spaces. 2. The multi-directional image input device according to claim 1, wherein the positional relationship detecting means corresponds to different focusing positions recorded in the standard sample image recording means for each of the plurality of optical systems. Image adding means for adding the image information of the plurality of standard samples, first peak position detecting means for detecting the peak position of the pixel density value from the added image information, and the first peak position detecting means. Second peak position detecting means for detecting a peak position from pixel density values of a plurality of standard sample images corresponding to different in-focus positions in the peak position detected by the above-mentioned arrangement of the plurality of optical systems. The relative positional relationship between the position and the peak position corresponding to each optical system obtained by the second peak detecting means is obtained, and the peak for each optical system obtained by the first and second peak position detecting means is obtained. Same position By determining to be in location, multidirectional image input apparatus and obtains the positional relationship of the standard sample observation image space each other corresponding to each of the plurality of optical systems. 3. A plurality of optical systems arranged to form an image of an object within a range of image input in a space for observing from different directions, and arranged on each image plane of the plurality of optical systems. In a multi-directional image input device having a plurality of imaging elements, a standard sample having a shape that can be set in the range of the image input where an object is present at the time of image input, and can be set as an optical reference when observing, Focusing position control means provided in each of the plurality of optical systems for moving a position (focusing position) of a focused object surface in the object space, and focusing of the optical system by the focusing position control means. Standard sample image recording means for recording the image information of the standard sample acquired by the image pickup device in a state where the position is set in a predetermined object space, and the information of the in-focus position at the time of acquiring the image information in association with each other; , The focusing position control hand While moving the in-focus position of the optical system by using the standard sample observation image space composed of images of the plurality of standard samples corresponding to different in-focus positions recorded in the standard sample image recording means, This standard sample observation image space corresponding to each of the systems, the positional relationship detecting means for detecting the positional relationship between the standard sample observation image space, and the standard sample observation image space corresponding to each of the plurality of optical systems detected by the positional relationship detecting means. An optical system position control means for controlling the spatial position of each of the plurality of optical systems based on the positional relationship, and a multidirectional image input device.