JP6111356B1 - Projection device - Google Patents

Abstract

An object of the present invention is to widen a range in which a projected image based on a captured image of a subject is accurately projected. A projection device (20) includes an imaging unit (210), a projection unit (220), a control unit (250), and an optical unit (201). The imaging unit captures a subject and generates a captured image. The projection unit irradiates the subject with projection light indicating a projection image based on the captured image. The control unit controls the projection image based on the captured image. The optical unit guides the optical axis (J1) of the incident light (310) incident on the imaging unit from the subject and the optical axis of the projection light (320) irradiated on the subject from the projection unit. . The angle of view (θc) captured by the imaging unit based on the incident light and the angle of view (θp) at which the projection unit emits the projection light match, and the optical path length (Lc) of the incident light between the imaging unit and the optical unit The optical path length (Lp) of the projection light between the projection unit and the optical unit is the same. [Selection] Figure 8B

Description

  The present disclosure relates to a projection apparatus that projects a projection image onto a subject.

  Patent Document 1 discloses an optical imaging system used in the medical field. The optical imaging system of Patent Document 1 includes an electronic imaging device that images a surgical field, a projector that projects a visible light image of an imaging result of the surgical field during surgery, and the optical axes of the electronic imaging device and the projector that are the same optical axis. An optical element. In Patent Document 1, the correspondence between a captured image and a projected image on the same optical axis is adjusted in advance before surgery using a transformation matrix that performs transformation such as enlargement, reduction, rotation, and translation on imaging data. Thus, calibration for accurately projecting a projection image at the time of surgery is performed.

US Patent Application Publication No. 2008/0004533

  An object of the present disclosure is to provide a projection device capable of widening a range in which a projection image based on a captured image of a subject is accurately projected.

  The projection device according to the present disclosure includes an imaging unit, a projection unit, a control unit, and an optical unit. The imaging unit captures a subject and generates a captured image. The projection unit irradiates the subject with projection light indicating a projection image based on the captured image. The control unit controls the projection image based on the captured image. The optical unit guides the optical axis of the incident light incident on the imaging unit from the subject and the optical axis of the projection light irradiated onto the subject from the projection unit. In the projection device, the angle of view captured by the imaging unit based on the incident light and the angle of view irradiated by the projection unit coincide with the optical path length of the incident light between the imaging unit and the optical unit, the projection unit, and the optical unit. The optical path length of the projection light between the parts matches.

  According to the projection device of the present disclosure, it is possible to widen a range in which a projection image based on a captured image of a subject is accurately projected.

Schematic which shows the structure of the surgery assistance system concerning Embodiment 1. FIG. The block diagram which shows the structure of the imaging irradiation apparatus in a surgery assistance system FIG. 3 is a plan view of the imaging irradiation apparatus according to the first embodiment. 1 is a side view of an imaging irradiation apparatus according to a first embodiment. Enlarged view of adjustment mechanism of dichroic mirror in imaging irradiation device Flowchart for explaining projection operation in surgery support system The figure for demonstrating the state of the operative field before projection operation | movement in a surgery assistance system The figure for demonstrating the state of the surgical field at the time of projection operation | movement in a surgery assistance system The figure for demonstrating position alignment of the surgery assistance system concerning Embodiment 1. FIG. The figure for demonstrating the relationship between a view angle and position shift in a surgery assistance system The figure for demonstrating the positioning according to the angle of view in a surgery assistance system The top view of the jig | tool in the positioning method concerning Embodiment 1. FIG. The figure for demonstrating the picked-up image of the jig | tool in the alignment method First display example of projected image projected by alignment method Second display example of projected image projected by alignment method Plan view of an imaging irradiation apparatus according to Modification 1 of Embodiment 1 The side view of the imaging irradiation apparatus concerning the modification 1 of Embodiment 1 The side view of the imaging irradiation apparatus concerning the modification 2 of Embodiment 1 Side view of the imaging irradiation apparatus according to the third modification of the first embodiment Flowchart for explaining alignment processing according to the second embodiment.

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

  The applicant provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and is not intended to limit the subject matter described in the claims. Absent.

(Embodiment 1)
1. Configuration 1-1. Outline of Surgery Support System An outline of a surgery support system including the projection apparatus according to the first embodiment will be described with reference to FIG. FIG. 1 is a schematic diagram illustrating a configuration of a surgery support system 100 according to the first embodiment.

  The surgery support system 100 includes a camera 210, a projector 220, and an excitation light source 230. The surgery support system 100 is a system that visually supports surgery performed by a doctor or the like on a patient in an operating room or the like using a projection image. When the surgery support system 100 is used, a photosensitive substance is administered in advance to the patient 120 undergoing surgery.

  A photosensitive substance is a substance that emits fluorescence in response to excitation light. Examples of the photosensitive substance include ICG (Indocyanine Green), 5-ALA (aminolevulinic acid), polyferrin, and the like. In this embodiment, a case where ICG is used as an example of a photosensitive substance will be described. The ICG emits fluorescence in the infrared region having a wavelength of 820 to 860 nm when irradiated with excitation light in the infrared region in the vicinity of a wavelength of 780 nm.

  When the photosensitive substance is administered to the patient 120, it accumulates in the affected area 130 where the flow of blood or lymph is stagnant. For this reason, it becomes possible to identify the region of the affected area 130 by detecting the region that emits fluorescence in response to the irradiation of the excitation light.

  Here, since the fluorescence emitted from the affected area 130 is weak or the wavelength band of the fluorescence is in the non-visible region or in the vicinity of the non-visible region, even if the doctor or the like visually observes the operative field 135, Is difficult to identify. Therefore, in the surgery support system 100, the excitation light 300 is irradiated from the excitation light source 230 to the surgical field 135, and the region of the affected part 130 that emits the fluorescence 310 is specified using the camera 210. Further, the projector 220 irradiates the affected part 130 with the projected light 320 of visible light so that the specified affected part 130 can be visually recognized by humans. Thereby, a projection image for visualizing the specified region of the affected part 130 is projected, and it is possible to support the specification of the region of the affected part 130 by a doctor or the like who performs the operation.

1-2. Configuration of Surgery Support System Hereinafter, the configuration of the surgery support system 100 will be described with reference to FIG. The surgery support system 100 is used by being placed in a hospital operating room. The surgery support system 100 includes an imaging irradiation device 200, a memory 240, and a projection control device 250.

  Although not shown, the surgery support system 100 includes a mechanism for changing the arrangement of the imaging irradiation apparatus 200, for example, a drive arm mechanically connected to the imaging irradiation apparatus 200, and a set of the surgery support system 100. Equipped with pedestal casters. With the above-described mechanism, the imaging irradiation apparatus 200 is disposed vertically above the operating table 110 on which the patient 120 is placed or at an angle from the vertical. The operating table 110 may include a drive mechanism that can change the height and direction.

  The imaging irradiation apparatus 200 is an apparatus in which a camera 210, a projector 220, and an excitation light source 230 are assembled together. As shown in FIG. 1, the imaging irradiation device 200 further includes a dichroic mirror 201 and a spacer 202. Details of the configuration of the imaging irradiation apparatus 200 will be described later.

  The memory 240 is a storage medium that is appropriately accessed when the projection control apparatus 250 executes various calculations. The memory 240 is composed of, for example, a ROM and a RAM.

  The projection control device 250 performs overall control of each part constituting the surgery support system 100. The projection control device 250 is electrically connected to the camera 210, the projector 220, the excitation light source 230, and the memory 240, and outputs a control signal for controlling each part. The projection control device 250 is constituted by a CPU, for example, and realizes its function by executing a predetermined program. Note that the function of the projection control apparatus 250 may be realized by a dedicated electronic circuit or a reconfigurable electronic circuit (FPGA, ASIC, or the like).

  For example, the projection control apparatus 250 performs various image processing on an image captured by the camera 210 to generate a video signal indicating the projection image. The projection control device 250 is an example of a control unit in the present disclosure, and the imaging irradiation device 200, the memory 240, and the projection control device 250 constitute the projection device 20 according to the present embodiment.

  In the present embodiment, the surgery support system 100 includes a display control device 150, a display 160, and a mouse 170. The display control device 150 is configured by a PC (personal computer), for example, and is connected to the projection control device 250.

  The operator 140 of the display control device 150 can check the image captured by the camera 210 on the display 160 during, for example, surgery. Further, the operator 140 can change various settings of the projection image (for example, a threshold value for the fluorescence intensity distribution).

1-3. Configuration of Imaging Irradiation Apparatus Next, details of the configuration of the imaging irradiation apparatus 200 will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the imaging irradiation apparatus 200 in the surgery support system.

  The excitation light source 230 is a light source device that irradiates excitation light 300 for causing a photosensitive substance to emit fluorescence. In this embodiment, since ICG is used as the photosensitive substance, the excitation light source 230 irradiates the excitation light 300 having a wavelength band (for example, 780 nm ± 30 nm) including the excitation wavelength of ICG. The excitation light source 230 switches on / off irradiation of the excitation light 300 in accordance with a control signal from the projection control device 250. The excitation light source 230 may be configured separately from the imaging irradiation device 200 (or the projection device 20).

  The camera 210 captures a subject such as the surgical field 135 of the patient 120 and generates a captured image. The camera 210 transmits the generated captured image to the projection control device 250. In the present embodiment, an infrared camera capable of imaging based on light in the wavelength band of 820 nm to 860 nm of fluorescence of ICG is used as the camera 210 together with the visible light region. The camera 210 is an example of an imaging unit in the present embodiment. As shown in FIG. 2, the camera 210 includes an image sensor 211, a telephoto lens 212, and an optical filter 213.

  The image sensor 211 is configured by, for example, a CCD image sensor or a CMOS image sensor. The imaging element 211 has an imaging surface on which light incident from the telephoto lens 212 forms an image.

  The telephoto lens 212 includes a zoom lens that sets the angle of view of the camera 210 and a focus lens that adjusts the focus. The telephoto lens 212 is an example of an imaging optical system in the camera 210. Instead of the telephoto lens 212, a standard lens, a medium telephoto lens, or a super telephoto lens may be used.

  The optical filter 213 includes a bandpass filter and a bandpass filter switching mechanism. As shown in FIG. 2, the optical filter 213 is disposed on the incident surface of the telephoto lens 212. The band-pass filter transmits a wavelength band component (for example, 850 nm ± 12.5 nm) in which fluorescence can be generated from a photosensitive material such as ICG among incident light, and blocks other wavelength band components. The switching mechanism switches insertion / extraction of the band pass filter with respect to the incident light to the telephoto lens 212 in accordance with on / off of the optical filter 213.

  The projector 220 is a projector such as a DLP system, a 3LCD system, or an LCOS system. The projector 220 emits projection light 315 so that a projection image based on the video signal input from the projection control device 250 is projected with visible light. The projector 220 is an example of a projection unit in the present embodiment. As shown in FIG. 2, the projector 220 includes a projection light source 221, an image forming unit 222, and a projection optical system 223.

  The projection light source 221 is configured by, for example, an LD (semiconductor laser), an LED, a halogen lamp, or the like. The projection light source 221 irradiates the image forming unit 222 with visible light. The projection light source 221 may include only a single color light source element, or may include a plurality of color light source elements such as RGB, or a white light source element, depending on the projection method of the projector 220.

  The image forming unit 222 includes a spatial light modulator such as DMD or LCD. The image forming unit 222 forms an image based on the video signal from the projection control device 250 on the image forming surface of the spatial light modulator. The light from the projection light source 221 is spatially modulated in accordance with the image formed on the image forming unit 222, whereby the projection light 315 is generated.

  Projection optical system 223 includes a zoom lens that sets the angle of view of projector 220 and a focus lens that adjusts the focus. The projection optical system 223 may incorporate a lens shift mechanism that shifts various lens positions.

  The projector 220 may include a projection control circuit that realizes functions unique to the projector 220 such as a keystone correction and a lens shift function. In addition, each function described above may be realized in the projection control apparatus 250.

  Further, the projector 220 may be a laser scanning type, and may be configured to include a MEMS mirror or a galvanometer mirror that can be driven in the scanning direction.

  The dichroic mirror 201 is an optical element having an optical characteristic of transmitting a specific wavelength band component of incident light while reflecting other wavelength band components. In the present embodiment, the dichroic mirror 201 transmits light having a wavelength band component greater than 650 nm (including ICG fluorescence) and reflects light having a wavelength band component smaller than 650 nm (including visible light). The dichroic mirror 201 is an example of an optical unit in the present embodiment. The optical characteristics of the optical unit can be set as appropriate according to the fluorescence characteristics of the photosensitive substance used.

  As shown in FIG. 2, the dichroic mirror 201 is disposed so as to face the camera 210 and the projector 220. The dichroic mirror 201 reflects the projection light 315 emitted from the projector 220 while transmitting the fluorescence 310 directed to the imaging surface of the camera 210 due to the optical characteristics described above. The reflected projection light 320 is irradiated on the surgical field 135.

  In the present embodiment, the dichroic mirror 201 has an optical axis of incident light that enters the camera 210 such as fluorescence 310 from the surgical field 135 and an optical axis of the projection light 320 that projects a projection image on the surgical field 135. The light is guided so as to coincide with the optical axis J1. Thereby, the position shift of the projection image based on the captured image of the camera 210 can be reduced.

  Note that an allowable error may be set as appropriate for the coincidence of the optical axes in the present disclosure. For example, the optical axes may coincide with each other within an allowable error within an angle range of ± 5 degrees or an optical axis interval of 1 cm.

1-4. Regarding Adjustment Mechanism In this embodiment, the imaging irradiation apparatus 200 includes various adjustment mechanisms for aligning the projected image. Hereinafter, the adjustment mechanism in the imaging irradiation apparatus 200 will be described with reference to FIGS. 3A, 3B and 4.

  3A and 3B are a plan view and a side view of the imaging irradiation apparatus 200 according to the present embodiment. FIG. 4 is an enlarged view of a mechanism for adjusting the rotational position of the dichroic mirror 201 in the imaging irradiation apparatus 200. 3A to 4, J <b> 2 indicates the optical axis of the projection light from the projector 220 before being reflected by the dichroic mirror 201. Hereinafter, the direction parallel to the optical axis J2 is referred to as “X direction”, the direction parallel to the optical axis J1 is referred to as “Z direction”, and the direction orthogonal to the optical axes J1 and J2 is referred to as “Y direction”.

  In the present embodiment, the projector 220 is configured to be shiftable in the Y direction as shown in FIG. 3A. The adjusting mechanism for shifting the projector 220 is configured by a long hole 220 a and a fixing screw 220 b provided in a portion where the projector 220 is assembled to the substrate 203. The longitudinal direction of the long hole 220a faces the Y-axis direction.

  By shifting the projector 220 along the long hole 220a, the projection position of the projection light reflected by the dichroic mirror 201 is shifted in the Y direction while maintaining the optical path length of the projection light from the projector 220 to the dichroic mirror 201. Can do. This mechanism is an example of a first adjustment unit in the present disclosure.

  In FIG. 3A, the housing of the projector 220 has a predetermined inclination (about 7 degrees) with respect to the optical axis J2. Not limited to this, the housing of the projector 220 may be parallel to the optical axis J2.

  In the present embodiment, the camera 210 and the dichroic mirror 201 are opposed to each other via the spacer 202 in the Z direction, as shown in FIG. 3B. The spacer 202 includes two lens barrels 202a and 202b and a fixing screw 202c.

  The lens barrel 202b is fixed to the dichroic mirror 201. The lens barrel 202a is engaged with the lens barrel 202b so as to be linearly movable in the Z direction. When the lens barrel 202a moves linearly, the distance between the camera 210 and the dichroic mirror 201 expands and contracts. The fixing screw 202c fixes the position of the lens barrel 202a with respect to the lens barrel 202b. With the spacer 202, the optical path length of the light incident on the camera 210 can be appropriately adjusted. The spacer 202 is an example of a second adjustment unit in the present embodiment.

  In the present embodiment, the dichroic mirror 201 is configured to be rotatable around a rotation axis 201a parallel to the Y-axis direction, as shown in FIG. 3B. The rotation axis 201a is located at the intersection of the optical axes J1 and J2. By rotating the dichroic mirror 201 with the rotation shaft 201a, the projection position of the projection light reflected by the dichroic mirror 201 can be shifted in the X direction while maintaining the optical path length of the projection light from the projector 220 to the dichroic mirror 201. it can.

  As shown in FIG. 4, the mechanism for adjusting the rotational position of the dichroic mirror 201 includes a rotation shaft 201a, a push screw 201b, a mirror fixing portion 201c, and lock screws 201d and 201e. The mirror fixing parts 201 c are provided on both sides of the dichroic mirror 201. The lock screws 201d and 201e are located on both sides of the rotation shaft 201a, and fix the mirror fixing portion 201c by tightening the screw, thereby restricting the rotation of the dichroic mirror 201. The push screw 201b presses the dichroic mirror 201 through the mirror fixing portion 201c from the one of the two lock screws 201d and 201e.

  Adjustment of the rotation angle of the dichroic mirror 201 is performed by changing the amount of pushing by the push screw 201b in a state where the lock screws 201d and 201e are loosened. Thereby, the rotation angle of the dichroic mirror 201 can be precisely adjusted.

  Further, in order to rotate the arrangement of the camera 210 on the XY plane, the lens barrel 202a may be configured to be rotatable with respect to the lens barrel 202b. In this case, the optical axis J1 of the camera 210 and the rotation axis of the lens barrel 202a are matched. The rotational position of the lens barrel 202a may be fixed by a fixing screw 202c. The mechanism for adjusting the rotational position of the dichroic mirror 201 and the mechanism for rotating the camera 210 on the XY plane are examples of the first adjustment unit in the present embodiment.

2. Operation Hereinafter, the operation of the surgery support system 100 according to the present embodiment will be described.

2-1. Basic Projection Operation of Surgery Support System The basic projection operation of the surgery support system 100 will be described with reference to FIGS. FIG. 5 is a flowchart for explaining a basic projection operation in the surgery support system 100. FIG. 6A shows the state of the surgical field 135 in the surgical operation support system 100 before performing the projection operation. FIG. 6B shows a state in which a projection operation is performed on the surgical field 135 of FIG. 6A.

  The flowchart in FIG. 5 is executed by the projection control apparatus 250. The process according to this flowchart is performed in the camera 210 with the optical filter 213 turned on.

  In the flowchart of FIG. 5, the projection control apparatus 250 first drives the excitation light source 230 to irradiate the surgical field 135 with the excitation light 300 as shown in FIG. 6A (S1). By irradiation with the excitation light 300, the affected area 130 in the operative field 135 emits fluorescence, and the fluorescence 310 from the affected area 130 enters the imaging irradiation apparatus 200.

  In the imaging irradiation apparatus 200, the fluorescence 310 passes through the dichroic mirror 201 and passes through the optical filter 213 of the camera 210, as shown in FIG. Thereby, the camera 210 receives the fluorescence 310 in the image sensor 211.

  Next, for example, the projection control apparatus 250 controls the camera 210 to image the operative field 135 and acquires a captured image from the camera 210 (S2). The captured image includes a fluorescence image based on the fluorescence 310 emitted from the affected part 130.

  Next, the projection control apparatus 250 performs image processing for generating a projection image based on the acquired captured image (S3). The projection control device 250 generates an image corresponding to the fluorescent image in the captured image and outputs it to the projector 220 as a video signal.

  In the image processing of step S3, for example, the projection control apparatus 250 binarizes the distribution of received light intensity in the captured image based on a predetermined threshold value, and extracts a fluorescent image region in the captured image. Next, the projection control apparatus 250 refers to various parameters stored in the memory 240, and performs coordinate conversion such as shift, rotation, enlargement / reduction, correction of image distortion, etc., for an image including the extracted area. I do. Thereby, the image showing the specific area | region according to the fluorescence image in a captured image is produced | generated.

  Next, the projection control device 250 controls the projector 220 so as to project a projection image based on the generated video signal (S4). Under the control of the projection control apparatus 250, an image corresponding to the video signal from the projection control apparatus 250 is formed on the image forming surface of the image forming unit 222 in the projector 220. The projector 220 drives the projection light source 221 so as to generate projection light 315 representing an image on the image forming surface with visible light, and emits the projection light 315 to the dichroic mirror 201 via the projection optical system 223 (FIG. 2). reference).

  As shown in FIG. 2, the dichroic mirror 201 reflects the projection light 315 that is visible light, and emits the projection light 320 along the optical axis J1. 6B, the imaging irradiation apparatus 200 irradiates the operation field 135 with the projection light 320, and the projection image G320 is projected onto the affected part 130 in the operation field 135. The projection image G320 is, for example, a single color image with one gradation.

  The above processing is repeatedly executed at a predetermined cycle (for example, 1/60 to 1/30 seconds).

  Through the above processing, the projection control device 250 identifies the region of the affected part 130 that emits fluorescence based on the captured image of the camera 210, and the projected image G320 of visible light is projected from the projector 220 onto the affected part 130. Thereby, in the surgery assistance system 100, the affected part 130 that is difficult to visually recognize can be visualized. The surgery support system 100 allows a doctor or the like to visually recognize the real-time state of the affected part 130.

  In the above description, an example in which the projection image G320 is a single color and one gradation image has been described. The projection control apparatus 250 may generate a multi-gradation projection image by determining the region of the fluorescent image in the captured image in multiple stages using, for example, a plurality of threshold values. Further, the projection control apparatus 250 may generate a projection image so as to continuously reproduce the distribution of received light intensity in the captured image. Further, the projection image is not limited to a single color, and may be generated in a plurality of colors or full colors.

2-2. Positioning operation 2-2-1. Overview As described above, the surgery support system 100 projects the projection image G320 from the projector 220 based on the captured image of the camera 210, thereby visualizing the affected area 130 that is difficult to visually recognize during surgery (see FIGS. 6A and 6B). . In the present system 100, in order to visualize the affected area 130 without error by the projection image G320 during use (during surgery), the image captured by the camera 210 and the projection image of the projector 220 are aligned in advance.

  FIG. 7 is a diagram for explaining alignment of the surgery support system 100 according to the present embodiment. FIG. 7 illustrates a situation where the worker 440 performs alignment in the surgery support system 100. The alignment is performed, for example, at the time of manufacturing, maintenance, and calibration before use.

  In FIG. 7, an information processing device 450 such as a PC is connected to the projection control device 250. In addition, a display 460, a mouse 470, and the like are connected to the information processing apparatus 450. In the example of FIG. 7, the worker 440 performs alignment by changing various parameters in the surgery support system 100 via the information processing device 450.

  Further, in the example of FIG. 7, the jig 500 is installed at the reference position P <b> 0 at the reference height where the subject (patient 120) is assumed to be located during the operation, and is positioned with reference to the jig 500. Matching is done. The Z direction parallel to the optical axis J1 is the height direction in the surgery support system 100.

  The surgery support system 100 is required to perform a projection operation while ensuring a wide range of motion in the height direction (Z direction). For example, as shown in FIG. 1, since the body of the patient 120 has a thickness, the projection image G320 during the operation (FIG. 2B) is accurately projected over a predetermined range (for example, 40 cm) in the height direction. It is necessary to In addition, the height of the operating table 110 and the imaging irradiation apparatus 200 may be changed due to circumstances such as changing the direction of the body of the patient 120 or changing the practitioner during the operation. Therefore, in the present embodiment, a projection device 20 that can widen the range in the height direction in which the projection image is accurately projected is provided.

2-2-2. In this embodiment, as shown in FIG. 7, alignment is performed so that the camera 210 and the projector 220 perform imaging or projection on the same optical axis J <b> 1 via the dichroic mirror 201. Done.

  As a result of intensive studies by the present inventor, even if alignment is performed while the optical axis of the camera 210 and the optical axis of the projector 220 are aligned with the same optical axis J1, a positional shift occurs depending on each angle of view. The relationship between the angle of view and the positional deviation will be described below with reference to FIG. 8A.

  FIG. 8A is a diagram for explaining the relationship between the angle of view and the position shift in the surgery support system 100. The reference height Z0 of the reference position P0 in alignment is appropriately set to a predetermined height in consideration of the operating state of the surgery support system 100. By performing alignment at the reference position P0, the entire area captured by the camera 210 matches the entire area that can be projected from the projector 220. As a result, the projected image can be accurately projected on the subject near the height Z0 based on the captured image.

  Here, in the above-described alignment, as shown in FIG. 8A, it is assumed that the angle of view θc ′ of the camera 210 and the angle of view θp ′ of the projector 220 do not match. In this case, if the height of the subject deviates from the reference height Z0, a positional deviation ΔD occurs between the captured image and the projected image according to the height deviation ΔZ. In this case, the range in which the projection image can be accurately projected based on the captured image is limited to the vicinity of the reference position Z0, and the movable range of the surgery support system 100 becomes narrow.

  Based on the above knowledge, the present inventor has intensively studied and has come up with a positioning method according to the present embodiment and a projection apparatus 20 that realizes the positioning method. Hereinafter, alignment by the projection apparatus 20 according to the present embodiment will be described with reference to FIG. 8B.

  In the present embodiment, as shown in FIG. 8B, in the alignment on the same optical axis J1 at the reference position Z0, the angle of view θc of the camera 210 and the angle of view θp of the projector 220 are matched. In addition, in order to realize alignment with the same angle of view θc = θp, the imaging irradiation apparatus 200 (projection apparatus 20) according to the present embodiment uses the spacer 202 to provide an optical path between the dichroic mirror 201 and the projector 220. The length Lp and the optical path length Lc between the camera 210 and the dichroic mirror 201 are matched.

  The optical path length Lp is, for example, the optical path length from the intersection P1 between the reflecting surface of the dichroic mirror 201 and the optical axis J1 to the principal point Pp of the projection optical system 223 in the projector 220. The optical path length Lc is, for example, the optical path length from the principal point Pc of the telephoto lens 212 (imaging optical system) in the camera 210 to the intersection point P1 on the dichroic mirror 201. The optical path lengths Lc and Lp are the optical characteristics of the camera 210 and the projector 220, the size of the imaging device 211, the size of the spatial light modulation device of the image forming unit 222, the height of the reference position P0, and the size of the region to be imaged (projected). It may be set as appropriate based on the above.

  As shown in FIG. 8B, by aligning at the same angle of view θp = θc on the same axis J1, the entire area imaged by the camera 210 and the projector 220 can be obtained even at heights Z1 and Z2 that deviate from the reference height Z0. The entire region that can be projected matches. Thereby, the range in which the projection image corresponding to the captured image can be accurately projected extends from the vicinity of the reference height Z0, and the range of motion of the surgery support system 100 can be expanded.

  In the present disclosure, the optical path length Lp = Lc (or the same angle of view θp = θc) is not necessarily intended to be exactly the same, and is exactly one as long as it is within a practical range. You don't have to. For example, if the optical path lengths Lp and Lc (or the angles of view θp and θc) match within a range of 90%, the optical path lengths (or angles of view) may match.

  Further, in the present embodiment, various adjustments are made to the imaging irradiation apparatus 200 so that the captured image and the projected image are aligned in the state of the coaxial J1, the same angle of view θp = θc, and the same optical path length Lp = Lc. A mechanism is provided. Hereinafter, an alignment method using the adjustment mechanism in the present embodiment will be described.

2-2-3. About the alignment method In the alignment method according to the present embodiment, coarse adjustment is performed using various adjustment mechanisms provided in the imaging irradiation apparatus 200, and further fine adjustment is performed close to the image processing in the projection control apparatus 250. Hereinafter, the alignment method according to the present embodiment will be described with reference to FIGS.

  As shown in FIG. 7, first, the worker 440 installs the jig 500 at the reference position P <b> 0 so that the optical axis J <b> 1 is positioned at the center of the jig 500. The height of the reference position P0 is the reference height Z0.

  FIG. 9A is a plan view of the jig 500 in the alignment method according to the present embodiment. As shown in FIG. 9A, the jig 500 is a frame-shaped member having an aspect ratio DY / DX. The dimensions DX and DY of the jig 500 are designed according to the reference height Z0 and the angle of view of the imaging irradiation apparatus 200 in the X direction and the Y direction. In this embodiment, the camera 210 and the projector 220 are set to have the same aspect ratio.

  Next, the worker 440 aligns the camera 210 with respect to the jig 500 at the reference position P0. The alignment of the camera 210 is performed based on the captured image of the jig 500 displayed on the display 460. FIG. 9B is a diagram for explaining a captured image Im1 of the jig 500 by the camera 210.

  FIG. 9B shows a state in which the inner frame of the jig 500 is reflected in the captured image Im1. For example, the worker 440 adjusts the zoom value of the telephoto lens 212 so that a captured image Im1 as illustrated in FIG. 9B is obtained. Here, when the rotational position of the jig 500 in the captured image Im1 is shifted, the rotational position of the camera 210 or the jig 500 is changed as appropriate.

  The worker 440 adjusts the focus of the camera 210 in a state where a captured image Im1 as shown in FIG. 9B is obtained.

  Next, the operator 440 aligns the projector 220 by using various adjustment mechanisms. The alignment of the projector 220 is performed using a projection image projected based on the captured image Im1. 10A and 10B are display examples of the projection image G500 projected at the reference position P0.

  The projection image G500 illustrated in FIGS. 10A and 10B is projected from the projector 220 to the reference position P0 by performing image processing by the projection control device 250 on the captured image Im1 in FIG. Further, the display 460 displays the state of the imaging region Rm1 captured in the captured image Im1.

  For example, the worker 440 adjusts the zoom value of the projection optical system 223 so that the size of the projection image G320 is equal to the size of the captured image Im1 based on the image displayed on the display 460, for example. Thereby, the angle of view of the camera 210 and the angle of view of the projector 220 can be substantially matched. Note that the operator 440 may adjust the zoom value while viewing the projection image G320 actually projected on the reference position P0.

  In the example of FIG. 10A, the orientation of the jig 500 (or the imaging region Rm1) and the projection image G500 is shifted. In such a case, the worker 440 aligns the directions of the jig 500 and the projection image G500 by rotating the camera 210 and the jig 500, for example. Then, as shown in FIG. 10B, the projection position of the projection image G500 is shifted with respect to the jig 500.

  The operator 440 adjusts the projection position of the projection image G320 in each of the X and Y directions in a state where the orientations of the jig 500 and the projection image G500 are aligned. Specifically, the worker 440 moves the projection image G500 in the X direction by rotating the dichroic mirror 201 around the rotation axis 201a (see FIG. 3B). In addition, the worker 440 moves the projection image G500 in the Y direction by shifting the projector 220 along the long hole 220a (see FIG. 3A).

  In addition, the worker 440 causes the projection control apparatus 250 to appropriately perform the keystone correction of the projection image G320 via the PC 450. When the operator 440 determines that the projection position of the projection image G500 matches the jig 500 within a predetermined error range (for example, 5 mm), the operator 440 matches the projection image G500 by the projector 220. The predetermined error is an error that is assumed that the projection image G320 cannot completely match the imaging region Rm1 by the adjustment by various adjustment mechanisms.

  In the alignment method according to the present embodiment, image processing is performed for a slight remaining error after the projection image G320 is matched with the imaging region Rm1 within a predetermined range by using various adjustment mechanisms as described above. Make fine adjustments. Specifically, the worker 440 determines the correction amount of the projection image G500 based on the captured image Im1 so that the projection image G500 accurately matches the captured region Rm1 on the display 460. The correction amount is, for example, an XY position shift amount, a rotation amount, a size ratio, a distortion correction value, or the like. The operator 440 causes the projection control apparatus 250 to store the determined various correction amounts in the memory 240 via the PC 450. The adjusted zoom value and the like are also stored in the memory 240.

  By the above method, the alignment of the coaxial J1 and the same angle of view θc = θp is performed in the optical path length Lc set by the spacer 202, and the range in the height direction in which the projected image is accurately projected can be widened. .

3. Effects etc. The projection device 20 in the above surgical support system 100 includes a camera 210, a projector 220, a projection control device 250, and a dichroic mirror 201. The camera 210 captures the affected area 130 of the operative field 135 that is a subject and generates a captured image. The projector 220 irradiates the subject with projection light 320 indicating a projection image based on the captured image. The projection control device 250 controls the projection image based on the captured image. The dichroic mirror 201 includes an optical axis J1 of incident light (fluorescence 310 emitted from the affected area 130) incident on the camera 210 from the surgical field 135, and an optical axis J1 of the projection light 320 irradiated onto the surgical field 135 from the projector 220. Are guided so that they match. In the projection device 20, the angle of view θc captured by the camera 210 based on the incident light coincides with the angle of view θp at which the projector 220 emits the projection light, and the optical path length Lc of the incident light between the camera 210 and the dichroic mirror 201 The optical path length Lp of the projection light 315 between the projector 220 and the dichroic mirror 201 matches.

  According to the projection device 20 described above, the optical path length Lc and the optical path length Lp coincide with each other in a state where the angle of view θc of the camera 210 and the angle of view θp of the projector 220 coincide on the same optical axis J1. For this reason, the positional deviation is reduced over a wide range in the direction of the optical axis J1, and the range in which the projection image based on the captured image of the subject is accurately projected can be widened.

  In addition, the projection device 20 according to the present embodiment includes various adjustment mechanisms 201a to 201c, 220a, and 220b (adjustable at least one of the positions and orientations of the camera 210, the projector 220, and the dichroic mirror 201). A first adjusting unit). This makes it easy to align the projected image with the coaxial J1, the same angle of view θc = θp, and the same optical path length Lc = Lp.

  In addition, the projection apparatus 20 according to the present embodiment further includes a spacer 202 configured so that the optical path length Lc of the incident light 310 between the camera 210 and the dichroic mirror 201 can be adjusted. The spacer 202 can align the projection images with the optical path lengths Lc and Lp matched.

  The spacer 202 does not have to be configured so that the optical path length Lc can be adjusted, and may be provided with a predetermined dimension (length in the Z direction). The predetermined dimension is a dimension such that the optical path length Lc = Lp in the state of the same angle of view θc = θp based on the optical characteristics of the optical systems 212 and 222. As a result, the optical path between the dichroic mirror 201 and the camera 210 and projector 220 is set so that the optical path lengths Lc and Lp match in the state of the same angle of view θc = θp.

  In this embodiment, the spacer 202 (second adjusting unit) adjusts the optical path length Lc by changing the distance between the camera 210 and the dichroic mirror 201. However, the distance is not limited to, for example, within the spacer. The optical path length Lc may be adjusted by providing an optical system. Further, the second adjustment unit in the present disclosure has a mechanism that can adjust the optical path length Lp of the projection light 320 between the projector 220 and the dichroic mirror 201 instead of or in addition to the spacer that adjusts the optical path length Lc. You may prepare. According to the spacer 202 for adjusting the optical path length Lc, when the projector 220 is larger than the camera 210, the imaging irradiation apparatus 200 can be made smaller than when a mechanism capable of adjusting the optical path length Lp of the projection light 320 is provided.

  Further, in the present embodiment, the first adjustment units 201a to 201c and 220a have the positions and orientations of the camera 210, the projector 220, and the dichroic mirror 201 while maintaining the optical path length Lc set by the spacer 202. At least one is configured to be adjustable. Thereby, it is possible to facilitate the alignment of the projected image in the state where the optical path length Lc = Lp.

  In the present embodiment, the first adjustment units 201a to 201c and 220a are configured to be able to move the projector 220 in the Y direction orthogonal to the X direction in which the projector 220 emits the projection light 315. Thereby, the shift of the projector 220 can be adjusted while maintaining the optical path length Lp to the dichroic mirror 201.

  In the present embodiment, the dichroic mirror 201 changes the traveling direction of the visible light component of the incident light and emits it. The first adjusting units 201a to 201c are arranged on a rotation axis 201a located at the intersection P1 of the optical axis J1 of the projection light 315 incident on the dichroic mirror 201 and the optical axis J2 of the projection light 320 emitted from the dichroic mirror 201. 201 is configured to be rotatable. Thereby, the emission direction of the projection light 320 can be adjusted while maintaining the optical path length Lp. Further, the emission direction of the projection light 320 can be adjusted only by a small mechanism that rotates the dichroic mirror 201.

  In the present embodiment, the projection control apparatus 250 adjusts the projection image G500 in the image processing based on the captured image Im1. As a result, it is possible to correct errors in the projected image that cannot be matched by rough adjustment by various adjustment mechanisms.

  In the present embodiment, the camera 210 includes a telephoto lens 212 that sets the angle of view θc of the camera 210. The projector 220 includes a projection optical system 223 that sets the angle of view θp of the projector 220. The projection control device 250 controls the telephoto lens 212 and the projection optical system 223 so that the angle of view θc and the angle of view θp coincide. The projection control device 250 can control the same angle of view θc = θp at various zoom values.

(Modification)
In Embodiment 1 described above, the position in the Y direction of the projection image is adjusted by the mechanism that shifts the projector 220 in the imaging irradiation apparatus 200, but the mechanism that adjusts the position in the Y direction is not limited to this. A modification of the mechanism for adjusting the position in the Y direction will be described with reference to FIGS. 11A and 11B.

  11A and 11B are a plan view and a side view of the imaging irradiation apparatus 200A according to the first modification of the first embodiment. As shown in FIG. 11A, the imaging irradiation apparatus 200A is configured to be able to rotate the projector 220 about the optical axis J1 on the XY plane. A mechanism for rotating the projector 220 on the XY plane includes, for example, a slider 205 provided between the projector 220 and the substrate 203 as shown in FIG. 11B.

  The slider 205 is a member that engages the projector 220 with the substrate 203 so as to be rotatable around the optical axis J1. By rotating the projector 220 by this mechanism, the projection position in the Y direction of the projection image can be adjusted while maintaining the optical path length of the projection light to the dichroic mirror 201.

  In Embodiment 1 described above, the dichroic mirror 201 is used as the optical unit in the imaging irradiation device 200. However, the optical unit is not limited to this, and other components that change the traveling direction of a specific component of incident light are used. It may be an optical element. A modification of the optical unit and its adjustment mechanism will be described with reference to FIGS. 12A and 12B.

  FIG. 12A is a side view of the imaging irradiation apparatus 200B according to the second modification of the first embodiment. The imaging irradiation device 200 </ b> B includes a prism 204 that refracts visible light as an optical unit that replaces the dichroic mirror 201.

  In this modification, the prism 204 is fixed, while the projector 220 is configured to be linearly movable in the Z direction. For example, as shown in FIG. 12A, the mechanism for moving the projector 220 linearly includes a support portion 205 </ b> A that can expand and contract in the Z direction between the projector 220 and the substrate 203. By directly moving the projector 220 with the support unit 205A, the position at which the projection light is emitted from the prism 204 is shifted, and the projection position in the X direction of the projection image can be adjusted.

  FIG. 12B is a plan view and a side view of the imaging irradiation apparatus 200C according to the third modification of the first embodiment. In the second modification, the projector 220 is shifted in the Z direction, but in this example, the projector 220 is rotated on the XZ plane with the intersection of the optical axes J1 and J2 as the rotation axis. A mechanism for rotating the projector 220 on the XZ plane includes a support portion 205B capable of setting an inclination, as shown in FIG. 12B, for example. By tilting the projector 220 with the support unit 205B, it is possible to adjust the projection position in the X direction of the projection image while maintaining the optical path length of the projection light to the prism 204.

  In Modifications 2 and 3, a mechanism for adjusting the projection position of the projection image in the Y direction may be provided as appropriate. For example, the projector 220 may be configured to be shiftable in the Y direction. It may be configured to be rotatable on the XY plane.

  In the second and third modifications, the prism 204 is fixed, but a mechanism for moving the prism 204 may be provided. For example, in order to adjust the projection position in the X direction of the projection image, the prism 204 may be configured to be rotatable on the XZ plane, or may be configured to be shiftable in the Z direction or the X direction.

  The adjustment mechanisms of the first to third modifications described above are examples of the first adjustment unit in the present disclosure. Each of the adjustment mechanisms 201a to 201c, 202, 220a, and 220b of the first embodiment and the adjustment mechanisms of the first to third modifications may each include a drive unit that can be automatically adjusted. The drive unit is composed of, for example, a linear motion unit and an actuator.

(Embodiment 2)
In the first embodiment, an example in which an operator performs alignment in the surgery support system 100 has been described. However, alignment in the surgery support system 100 may be automatically performed by various control devices. In the second embodiment, an example in which the projection control apparatus 250 performs alignment will be described.

  Hereinafter, description of the same configuration and operation as those of the surgery support system 100 according to the first embodiment will be omitted as appropriate, and the surgery support system 100 according to the present embodiment will be described.

  In the present embodiment, as an example, the drive unit is incorporated in the various adjustment mechanisms 201a to 201c, 202, 220a, and 220b (see FIGS. 3A, 3B, and 4) described in the first embodiment, and the projection control apparatus 250 performs adjustment. It is assumed that the mechanism can be driven.

  FIG. 13 is a flowchart for explaining the alignment processing according to the present embodiment. The alignment process is a process in which the projection control apparatus 250 aligns the projection image by driving various adjustment mechanisms based on the captured image of the camera 210. In the present embodiment, the process shown in FIG. 13 is executed by the projection control apparatus 250 and is started in a state where the jig 500 is installed at the reference position P0 (see FIGS. 7 and 9A).

  First, the projection control apparatus 250 causes the camera 210 to image the jig 500 at the reference position P0, and sets the angle of view of the camera 210 based on the captured image (S11). For example, the projection control apparatus 250 adjusts the zoom value of the camera 210 so that the inner frame of the jig 500 appears in the captured image Im1 (see FIG. 9B).

  Next, the projection control apparatus 250 projects the projection image G500 based on the captured image Im1, and sets the angle of view of the projector 220 based on the imaging result of the projection image G500 (S12). For example, the projection control apparatus 250 calculates the size of the projection image G500 based on the imaging result of the projection image G500, and adjusts the zoom value of the projector 220 according to the calculation result. Further, the projection control device 250 may set the zoom value of the projector 220 based on the adjusted zoom value of the camera 210.

  Next, based on the imaging result of the projection image G500 in which the angle of view is set, the projection control device 250 makes the projection image G500 coincide with the imaging region Rm1 within a predetermined error range (see FIGS. 10A and 10B). Various adjustment mechanisms are driven (S13). For example, the projection control apparatus 250 rotates the dichroic mirror 201 to move the projection image G500 in the X direction, or shifts the projector 220 to move the projection image G500 in the Y direction.

  Next, the projection control apparatus 250 performs fine adjustment in image processing on the error that remains slightly in the coarse adjustment by driving the various adjustment mechanisms as described above (S14). Specifically, the projection control apparatus 250 calculates the correction amount of the projection image G500 based on the captured image Im1 so that the projection image G500 exactly matches the imaging region Rm1.

  Next, for example, the projection control apparatus 250 stores various adjusted parameters such as the zoom values of the camera 210 and the projector 220 and the calculated correction amount in the memory 240 (S15), and ends this processing.

  Through the above processing, the projection control apparatus 250 can automatically perform alignment based on the imaging result of the camera 210.

  As described above, in the surgery support system 100 according to the present embodiment, the projection control apparatus 250 drives the first adjustment unit so as to adjust the projection image G500 on the subject based on the captured image Im1. Thereby, the alignment in the surgery assistance system 100 can be performed automatically.

  In the above description, the first adjustment unit is driven to move the projection image G500. The spacer 202 (second adjustment unit) may be driven to adjust the optical path length as appropriate in accordance with the zoom or focus change of the camera 210 and the projector 220, for example, without being limited to the first adjustment unit.

(Other embodiments)
As described above, Embodiments 1 and 2 have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can also be applied to an embodiment in which changes, replacements, additions, omissions, and the like are appropriately performed. In addition, it is possible to combine the components described in the first and second embodiments to form a new embodiment.

  Therefore, other embodiments will be exemplified below.

  In the first and second embodiments, the dichroic mirror 201 or the prism 204 that transmits the fluorescence generated from the photosensitive substance has been described. However, the present invention is not limited thereto, and the dichroic that reflects or refracts the fluorescence is used as an optical unit in the present disclosure. A mirror or prism may be used. In this case, the arrangement of the imaging unit and the projection unit may be switched as appropriate. In addition, as an optical unit in the present disclosure, a polarizing plate that changes a traveling direction of light of a predetermined polarization component may be used. In this case, the light can be selectively incident on the imaging unit or the projection unit according to the polarization component of the light.

  In the first and second embodiments, medical applications such as surgery have been described as examples, but the present invention is not limited to this. For example, the present invention can be applied when it is necessary to work on an object whose state change cannot be visually confirmed, such as a construction site, a mining site, a construction site, or a factory that processes materials.

  Specifically, instead of the medical device of the first embodiment, a fluorescent material is applied to an object that cannot be visually confirmed in a construction site, a mining site, a construction site, or a factory that processes the material. Alternatively, it may be a subject to be imaged by the camera 210 by being kneaded or poured. Instead of light emission, a heat generation location may be detected by a thermal sensor, and only that portion or only the boundary may be scanned.

  As described above, the embodiments have been described as examples of the technology in the present disclosure. For this purpose, the accompanying drawings and detailed description are provided.

  Accordingly, among the components described in the accompanying drawings and the detailed description, not only the components essential for solving the problem, but also the components not essential for solving the problem in order to illustrate the above technique. May also be included. Therefore, it should not be immediately recognized that these non-essential components are essential as those non-essential components are described in the accompanying drawings and detailed description.

  Moreover, since the above-mentioned embodiment is for demonstrating the technique in this indication, a various change, substitution, addition, abbreviation, etc. can be performed in a claim or its equivalent range.

  The projection system according to the present disclosure can be applied to work on an object that is difficult to visually confirm a change in state, such as a medical use, a construction site, a mining site, a construction site, or a factory that processes materials. .

DESCRIPTION OF SYMBOLS 100 Surgical support system 200 Imaging irradiation apparatus 201 Dichroic mirror 202 Spacer 204 Prism 210 Camera 212 Telephoto lens 220 Projector 223 Projection optical system 250 Projection control apparatus 20 Projection apparatus

Claims (8)

An imaging unit that images a subject and generates a captured image;
A projection unit that irradiates the subject with projection light indicating a projection image based on the captured image;
A control unit for controlling the projection image based on the captured image;
An optical unit that guides an optical axis of incident light that is incident on the imaging unit from the subject and an optical axis of projection light that is projected onto the subject from the projection unit;
The angle of view captured by the imaging unit based on the incident light matches the angle of view of the projection unit irradiating the projection light,
The optical path length of the incident light between the imaging unit and the optical unit is the same as the optical path length of the projection light between the projection unit and the optical unit ,
A first adjustment unit configured to be capable of adjusting at least one of the position and orientation of each of the imaging unit, the projection unit, and the optical unit;
The optical unit changes the traveling direction of a predetermined component of incident light and emits it,
The first adjustment unit is configured to be able to rotate the optical unit on a rotation axis located at an intersection of an optical axis of light incident on the optical unit and an optical axis of light emitted from the optical unit. /> Projection device. The imaging unit with claim 1, further comprising a second adjustment unit adjustably configured at least one of the optical path length of the projected light between the optical path length and the projecting portion and the optical portion of the incident light between the optical portion The projection apparatus described in 1. The first adjustment unit is configured to be capable of adjusting at least one of the positions and orientations of the imaging unit, the projection unit, and the optical unit while maintaining the optical path length adjusted by the second adjustment unit. The projection apparatus according to claim 2 . The first adjustment unit, the projection apparatus according to any one of movable in claims 1 to 3, the projection portion in the direction in which the projecting portion is perpendicular to the direction for emitting the projection light. The control unit, on the basis of the captured image, to adjust the projected image on the object, a projection apparatus according to any one of claims 1 to 4 for driving the first adjusting portion. Wherein the control unit, the projection apparatus according to any one of claims 1 to 5, to adjust the projected image in the image processing based on the captured image. The imaging unit includes an imaging optical system that sets an angle of view of the imaging unit,
The projection unit includes a projection optical system that sets an angle of view of the projection unit,
Wherein the control unit includes, as the angle of view of the imaging unit and the angle of view of the projection unit are matched, the projection apparatus according to any one of claims 1 to 6 for controlling the imaging optical system and the projection optical system . Said optical unit, a projection apparatus according to any one of constituted claims 1-7 dichroic mirror or prism.

Images