JP2013205885A - Display device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform input to a screen displayed in a space.SOLUTION: A display device (1) comprises an image creating device (2) that creates an image (V), and an element surface (S) that is inclined to an image creation surface (2a) of the image creating device and on which light from an image is made incident. The display device further includes an imaging element (4) that forms an image (M) on a real image surface (J) that is plane symmetric to the image creation surface with respective to the element surface, an illumination unit that illuminates an illumination area including the real image surface, a camera (3) that photographs the illumination area, and a detection unit (6) that detects an indicator indicating the real image surface on the basis of the photographic image photographed by the camera.

Description

  The present invention relates to a display device.

  Conventionally, a non-contact input device has been proposed in which a screen displayed in a space such as a head-up display is designated with a fingertip to perform an input operation (for example, see Patent Document 1 below). In Patent Document 1, infrared light is scanned in a two-dimensional direction, an infrared sensor detects that the indicator placed in the space is irradiated with infrared light, and the position in the space is obtained, and the corresponding display surface An input device for positioning a predetermined location above is described.

  A technique for realizing a non-contact touch position input device with a finger in front of a screen displayed by a projector has also been proposed (see, for example, Patent Document 2 below). In Patent Document 2, a screen projected on a screen is photographed, and a touch position is detected based on a difference between a projection original image of a projector and a photographed image from a camera.

JP-A-8-202480 JP 2011-118533 A

  The input device of Patent Literature 1 as described above requires a complicated mechanism and processing for tracking while detecting a scanning mirror for scanning infrared rays. Further, the touch input device of Patent Document 2 cannot detect the touch position in principle unless a screen is present. As described above, in the conventional technique, in order to enable input using a screen displayed in a space, there is a disadvantage that the apparatus becomes large. The present invention has been made in view of the above-described circumstances, and an object thereof is to provide a display device that can satisfactorily input a screen displayed in a space.

  The display device according to the first aspect of the present invention includes an image generation device that generates an image, and an element surface that is inclined with respect to the image generation surface of the image generation device and on which light from the image is incident, An imaging element that forms an image on a real image plane that is plane-symmetric with the image generation plane with respect to the element plane, an illumination unit that illuminates an illumination area that includes the real image plane, a camera that captures the illumination area, and the camera And a detection unit that detects an indicator that indicates the real image plane based on the captured image.

  In the display device of the first aspect, the vicinity of the image formed by the imaging element is illuminated by the illumination unit and photographed by the camera, so that the indicator touching the image is illuminated and photographed, and input using the indicator is performed. It can be easily detected.

  In the display device according to the first aspect, the illumination unit has a first illumination state and a second illumination state, and associates the timing of acquiring a captured image of the camera with the illumination state of the illumination unit. A control unit that controls one or both of the timing of acquiring the captured image and the illumination state; and the detection unit captures the captured image captured by the camera in the first illumination state, and the second illumination state. The indicator may be detected by comparing with a captured image captured by the camera.

  Since this display device compares photographed images with different illumination states, the background other than the indicator in the photographed image can be removed, and the indicator can be accurately detected.

  In the display device according to the first aspect, the illumination unit may be disposed on the same side as the image generation surface with respect to the element surface, and illuminate the illumination region via the imaging element.

  This display device can illuminate by utilizing the light condensing action of the imaging element, and the device configuration can be simplified. Further, in this display device, since the illumination unit is disposed on the side opposite to the side on which the image is formed with respect to the element surface, the number of members disposed on the side on which the image is formed on the element surface is reduced. Therefore, it is avoided that such a member hinders image display.

  In the display device according to the first aspect, the illumination unit may be provided so that illumination light propagates along the real image plane.

  This display device can accurately associate the position of the indicator illuminated by the illumination light with the position on the real image plane.

  In the display device according to the first aspect, the illumination unit includes a plurality of solid light sources arranged in a predetermined direction, and emits spot-shaped illumination light that is emitted from the plurality of solid light sources and has the predetermined direction as a longitudinal direction. It may be provided so that the illumination light is limited in the normal direction of the real image plane.

  In this display device, the spread of the illumination light is limited in the normal direction of the real image plane, so that the indicator illuminated by the illumination light can be accurately detected.

  In the display device according to the first aspect, the shooting range on the real image plane of the camera may include the entire range in which the imaging element can form the image on the real image plane.

  This display device can detect a pointer regardless of the position of the pointer on the real image plane, and thus can support various inputs.

  The display device of the first aspect includes a plurality of processing units including the image generation device and the camera paired with the image generation device, and the imaging elements cross each other in the first direction. A first set of reflective surfaces, and a second set of reflective surfaces that cross each other in the second direction, and the image generation device of the first processing unit includes the first set of reflective surfaces as the first set of reflective surfaces. The image generation device of the second processing unit may be arranged with the image generation surface facing the second set of reflection surfaces.

  Since this display device can accept input from a plurality of directions while displaying images in a plurality of directions, it can cope with a plurality of users, for example. In addition, since the imaging device is shared by a plurality of processing units in this display device, the number of parts can be reduced.

  In the display device according to the first aspect, the image generation device may include a flat panel display having the image generation surface as a screen.

  This display device can display an image displayed on the screen of a flat panel display by forming an image on a real image plane and reduce the device cost.

  In the display device according to the first aspect, the image generation device may include a screen disposed on the image generation surface and a projector that projects the image onto the screen.

  This display device can display an image projected by a projector on a screen by forming an image on a real image surface and display a bright image.

It is a perspective view which shows typically the monitoring apparatus in 1st Embodiment. It is a side view which shows a monitoring apparatus typically. (A) is a figure which shows the example of a retrotransmissive material, (B) is an enlarged view of (A). (A)-(E) are figures which show the principle of the image formation by a retrotransmissive material. It is a 2nd page figure which shows the structure of an illumination part. It is a conceptual diagram which shows the detection method of a pointer. It is a block diagram which shows the function structure of a control apparatus. It is a timing chart which shows the timing of control by a control device. It is a conceptual diagram which shows an example of the input method. It is a conceptual diagram which shows the other example of the input method. It is a figure which shows the modification 1 of a retrotransmissive material. It is a figure which shows the modification 2 of a retrotransmissive material. It is a figure which shows the display apparatus in 2nd Embodiment. It is a figure which shows the display apparatus in 3rd Embodiment.

  Next, an embodiment will be described. In the embodiment, the same elements are denoted by the same reference numerals, and the description thereof may be simplified or omitted.

[First Embodiment]
FIG. 1 is a perspective view schematically showing a schematic configuration of a display device 1 of the present embodiment. FIG. 2 is a side view showing the display device 1 of the present embodiment. FIG. 3 is a diagram showing the configuration of the retropermeable material 4, FIG. 3A is a plan view seen from the thickness direction of the retropermeable material 4, and FIG. It is a perspective view which expands and shows a part.

  As shown in FIGS. 1 and 2, the display device 1 of the present embodiment includes a flat panel display 2 (image generation device), a camera 3 (imaging device), a retrotransmissive material 4 (imaging element), and an illumination unit 5. (Shown in FIG. 2) and a control device 6. The display device 1 displays an image (real image M) by the flat panel display 2 and the retrotransmissive material 4, images an indicator such as a finger of the observer K by the camera 3, and detects the indicator to display the image. An input to the device 1 is received.

  The flat panel display 2 (hereinafter abbreviated as FPD) displays an image V on an image display surface 2a (image generation surface) based on image data input from the control device 6. The image V includes, for example, an image showing a person, an image showing a landscape, an image showing characters, and the like. The FPD 2 is configured by a general display such as a liquid crystal display, a plasma display, and an organic EL display.

  As shown in FIGS. 3 (A) and 3 (B), the retrotransmissive material 4 has a plurality of rectangular column shapes through which light passes (transmits) through a plate material 7 such as rectangular glass having a predetermined thickness. The opening 8 is provided. The plate member 7 has two main surfaces 7a and 7b substantially parallel to each other, and light incident from one main surface 7a side is emitted from the other main surface 7b side.

  In the present embodiment, the positional relationship of each part of the display device 1 may be described with reference to the XYZ orthogonal coordinate system shown in FIG. In this XYZ orthogonal coordinate system, two directions orthogonal to each other on the element surface S of the retrotransmissive material 4 are the X direction and the Y direction, and the thickness direction of the retrotransmissive material 4 (the normal direction of the element surface S) is Z direction. Moreover, the V1 direction shown in FIG.1 and FIG.2 is a direction which cross | intersects each of X direction and Y direction in XY plane, for example, is a diagonal direction of the rectangular-plate-shaped retrotransmissive material 4. FIG.

  The element surface S is a plane parallel to the main surface 7a and the main surface 7b of the plate 7 and passes through the center of the plate 7 (the distance from the main surface 7a is the same as the distance from the main surface 7b). Flat surface. The element surface S may be a plane that passes through the center in the thickness direction of the plate material 7 and is orthogonal to the thickness direction (Z direction) of the plate material 7.

  The openings 8 are repeatedly arranged in two directions (X direction and Y direction) perpendicular to each other on the plate material 7. The planar shape of the opening 8 viewed from the normal direction of the element surface S is, for example, a square. In the retrotransmissive material 4 of the present embodiment, when viewed from the thickness direction (Z direction) of the plate material 7 (in plan view), each side of the opening 8 is substantially parallel to each side of the plate material 7.

  Thus, the retrotransmissive material 4 has a periodic structure (opening 8) in each of two directions (here, the X direction and the Y direction), and the element surface S has the retrotransmissive material. It is good also as a virtual plane inside 4 and a surface parallel to each of two directions where a periodic structure is located.

  The opening 8 is configured to function as a light passing part through which light passes. The opening 8 may be a space (gap) inside, or may be filled with a highly transparent resin material or the like.

  Two inner wall surfaces orthogonal to each other among the four inner wall surfaces of the opening 8 are reflection surfaces 9 (first reflection surface 9a and second reflection surface 9b). The reflecting surface 9 is made of, for example, a metal reflecting film formed in the opening 8.

  The 1st reflective surface 9a and the 2nd reflective surface 9b comprise what is called a 2 surface corner reflector. The first reflecting surface 9a and the second reflecting surface 9b face any direction between the + X direction and the + Y direction (first quadrant) with the center of each opening 8 as the origin in any of the plurality of openings 8. ing.

    The configuration in which two inner wall surfaces orthogonal to each other among the inner wall surfaces of the opening 8 are reflection surfaces has been described. However, all four inner wall surfaces of the opening 8 may be reflection surfaces 9, and in the present embodiment, The four inner wall surfaces of the opening 8 are all reflective surfaces 9. Of the reflecting surfaces 9, the third reflecting surface 9c is orthogonal to the second reflecting surface 9b and faces the first reflecting surface 9a substantially in parallel. Of the reflecting surfaces 9, the fourth reflecting surface 9d is orthogonal to the third reflecting surface 9c and faces the second reflecting surface 9b substantially in parallel.

  A pair of reflecting surfaces that intersect and are adjacent to each other in the reflecting surface 9 face the same direction, and light can enter from this direction. For example, the second reflecting surface 9b and the third reflecting surface 9c face an arbitrary direction between the + X direction and the -Y direction (fourth quadrant). Further, the third reflecting surface 9c and the fourth reflecting surface 9d face an arbitrary direction between the −X direction and the −Y direction (third quadrant). Further, the fourth reflecting surface 9d and the first reflecting surface 9a face an arbitrary direction between the −X direction and the + Y direction (second quadrant).

  As shown in FIG. 2, when viewed from the side parallel to the element surface S and the image display surface 2a, the FPD 2 is configured such that the image display surface 2a is inclined non-perpendicular to the element surface S of the retrotransmissive material 4. Have been placed.

  Further, as shown in FIG. 3A, when viewed from a direction perpendicular to the element surface S of the retrotransmissive material 4, the FPD 2 has a normal direction 2b of the image display surface 2a that is an arrangement direction (X (Direction, Y direction). In other words, when the FPD 2 is viewed from the direction perpendicular to the element surface S of the retrotransmissive material 4, the normal direction of the image display surface 2 a is an angle at which the two reflecting surfaces 9 of the opening 8 of the retrotransmissive material 4 contact each other. It is inclined and arranged so as to face the part side.

  Due to the positional relationship between the FPD 2 and the retrotransmissive material 4, the light emitted from the image display surface 2 a of the FPD 2 is approximately in the direction of the corner where the two reflecting surfaces 9 of the opening 8 of the retrotransmissive material 4 are in contact. Proceed toward. In FIG. 1, the illustration of the opening 8 of the retrotransmissive material 4 is omitted for easy understanding of the drawing. The position of the corner where the two reflecting surfaces 9 of the opening 8 of the retrotransmissive material 4 contact corresponds to the position of the corner indicated by R in FIG. To do.

  Here, the effect | action of the retrotransmissive material 4 is demonstrated using FIG. 4 (A)-(E). In FIG. 4, a point P indicates a point at which light on the image V illustrated in FIG. 2 is emitted, for example, each pixel on the FPD 2. A point T indicates a point (incident position) where light emitted from the point P enters the reflecting surface 9 of the retrotransmissive material 4. A point Q indicates a point where light emitted from the point P and passing through the retrotransmissive material 4 is imaged. 4D and 4E are a plan view and a side view, respectively, in which the vicinity of the point T is enlarged.

  As described above, the two reflecting surfaces 9 (the first reflecting surface 9a and the second reflecting surface 9b) of the retrotransmissive material 4 are orthogonal to each other. Therefore, as shown in FIG. 4D, the light incident on the second reflecting surface 9b is reflected by the second reflecting surface 9b, then enters the first reflecting surface 9a, and is reflected by the first reflecting surface 9a. , It proceeds in the direction opposite to the direction when it enters the second reflecting surface 9b. Therefore, the light incident on the reflecting surface 9 is the same as the incident direction as seen from a direction (Z direction) orthogonal to the element surface S as shown in FIG. Reflect in the direction. For this reason, the projected point obtained by projecting the point Q on the XY plane substantially coincides with the projected point obtained by projecting the point P on the XY plane.

  Further, the light incident on the reflecting surface 9 is in the normal direction of the incident surface with respect to the retrotransmissive material 4 (the normal direction of the triangle PTQ formed by the points P, T, and Q) as shown in FIG. From the perspective, the light is reflected by the first reflecting surface 9a and the second reflecting surface 9b. Therefore, as shown in FIG. 4C, the light incident on the reflecting surface 9 is reflected at the same angle as the incident angle a, equivalent to the case where a plane mirror is placed so as to be orthogonal to the incident surface and the element surface S. Reflects at angle a.

  Thus, the light emitted from the point P goes to the point Q through the point T as shown in FIG. Generally, the light emitted from the point P on the image V is diffused within a certain range of angles, so that the light is also incident on portions other than the point T of the retrotransmissive material 4. Since the light diffused from the point P is similarly reflected at locations other than the point T, it is focused on the point Q. That is, the image V located on one side of the element surface S is imaged at a plane symmetrical position (real image plane J) with respect to the element surface S in the space on the other side of the element surface S.

  Here, the light emitted from one point of the planar image V from P has been described, but the light emitted from each point on a three-dimensional object or image having a finite size is similarly imaged. Therefore, when a three-dimensional object exists below the retrotransmissive material 4, a real image that is plane-symmetric with respect to the element surface S of the retrotransmissive material 4 is formed as a three-dimensional image around the point Q.

  When observing the point Q on the real image, the light rays from the straight line connecting the observation point and the point Q are selectively viewed, and the real image appears to float in space when observed with both eyes. Further, as described above, since the retrotransmissive material 4 has the opening 8 that penetrates in the direction perpendicular to the element surface S, the retrotransmissive material 4 is incident on the element surface S of the retrotransmissive material 4 perpendicularly. The light travels straight without entering the reflecting surface 9.

  In the present embodiment, as shown in FIG. 2, the FPD 2 is inclined and installed below the element surface S of the retrotransmissive material 4. Therefore, the real image surface J is disposed in a space above the element surface S of the retrotransmissive material 4 in a direction symmetrical to the inclination of the image display surface 2 a of the FPD 2. The retrotransmissive material 4 forms, on such a real image surface J, a real image M in which the image V on the image display surface 2a is plane-symmetric. In this case, since the image display surface 2a of the FPD 2 is a flat surface, the formed real image M is different from the image V in inclination and is a flat image having substantially the same shape and dimensions.

  Therefore, if the observer K (user) looks into the direction of the real image M obliquely from the direction in which the real image plane J is inclined, the real image M is in a substantially opposite state to the observer K. Makes it easier to see. Since the real image M is formed by the light reflected by the retrotransmissive material 4, the observer K seems to have floated in the space within the observation range where the retrotransmissive material 4 can be seen. Looks like.

  Next, the camera 3 will be described. The camera 3 (photographing device) includes an image sensor such as a CCD sensor and a CMOS sensor, a photographing lens, an autofocus mechanism, and the like, and can be configured by, for example, a general camera. In the camera 3, the shooting timing is controlled by the control device 6 illustrated in FIG. 1, and the shot image data indicating the shot image is output to the control device 6.

  As shown in FIG. 2, the camera 3 is arranged on the side where light enters from the retrotransmissive material 4 with respect to the real image plane J. That is, the camera 3 is arranged so that the real image plane J is sandwiched between the camera 3 and the viewpoint (observer K) that observes the real image M. Thereby, for example, when the observer K observes the real image M formed on the real image plane J from the front, the observer K sees the camera 3 behind the real image M.

  At least a part of the camera 3 is disposed below the real image plane J, and is fixed to the retrotransmissive material 4, for example. The camera 3 is disposed so as to look upward from the element surface S. That is, the photographing direction Dp (incident side optical axis of the photographing lens) of the camera 3 is inclined so as to approach the real image surface J from the element surface S. In this case, the shooting direction Dp of the camera 3 connects the viewpoint (observer K) observing the real image M and a point on the real image M, as compared with the case where the camera 3 is disposed above the real image plane J. It becomes nearly parallel to the line (line of sight Kv). The shooting range 3a on the real image plane J of the camera 3 is set so as to include at least a part of the image forming range where the real image M is formed on the real image plane J. Here, the photographing range 3a is set so as to include almost the entire image forming range.

  Further, as shown in FIG. 3A, the camera 3 is arranged so that the photographing direction Dp passes through a real image forming range Ja in which a real image M on the real image plane J can be formed. For example, the shooting direction Dp of the camera 3 is set substantially coaxially with the normal direction 2b of the image display surface 2a when viewed from the normal direction of the element surface S. Note that the real image formation range Ja is, for example, a range optically conjugate with the range in which the image V is displayed on the image display surface 2a of the FPD 2.

  Next, the illumination unit 5 will be described with reference to FIGS. 2 and 5. The illumination unit 5 is controlled by the control device 6 (see FIG. 1) and illuminates the illumination area A1 including the real image plane J. The camera 3 captures an area that can be illuminated by the illumination unit 5 (illumination area A1).

  As shown in FIG. 2, the illumination unit 5 is arranged on the same side as the image display surface 2 a with respect to the retrotransmissive material 4, and the illumination area A <b> 1 is defined through the opening 8 of the retrotransmissive material 4. Illuminate. The illumination unit 5 of the present embodiment is attached to the top of the FPD 2 and extends along a side parallel to the element surface S out of the two sides of the rectangular image display surface 2a. The illumination light emitted from the illumination unit 5 is reflected by the reflection surface 9 of the retrotransmissive material 4 and propagates through a spatial region (illumination region A1) including the real image surface J.

  In the present embodiment, the illumination light emitted by the illumination unit 5 is pattern light in which the spot shape on the surface orthogonal to the emission direction from the retrotransmissive material 4 has a longitudinal direction and a short direction. Such pattern light may be called sheet light, line light, or the like.

  FIG. 5 is a two-view diagram illustrating the configuration of the illuminating unit 5, and corresponds to a side view of a side with respect to the emission direction of the illumination light from the illuminating unit 5. Specifically, FIG. 5A is a diagram illustrating a configuration in the longitudinal direction of the illuminating unit 5, and corresponds to a side view in a plane orthogonal to the Z-V1 plane in FIG. FIG. 5B is a diagram showing the structure of the illumination unit 5 in the short direction, and corresponds to a side view orthogonal to FIG.

  As shown in FIG. 5A, the illumination unit 5 includes a plurality of solid light sources 15, a cylindrical lens 16 on which illumination light emitted from the plurality of solid light sources 15 is incident, and a drive circuit that drives the plurality of solid light sources 15. 17. The drive circuit 17 is connected to the control device 6 and drives the plurality of solid state light sources 15 in accordance with a control command from the control device 6.

  The plurality of solid light sources 15 are arranged one-dimensionally in a predetermined direction. The predetermined direction is a direction (longitudinal direction) substantially parallel to each of the image display surface 2a and the element surface S. The solid light sources 15 may be two-dimensionally arranged. In this case, one of the two arrangement directions may be the above-described longitudinal direction.

  The solid light source 15 is composed of, for example, an LED (light emitting diode), an LD (laser diode), or the like. The solid light source 15 emits, for example, infrared light with a light amount corresponding to the supplied current as illumination light. The drive circuit 17 can supply current to the solid light source 15 to turn on the solid light source 15, and can stop supply of current to the solid light source 15 to turn off the solid light source.

  The control device 6 controls the light emission state of the solid light source 15 (the illumination state of the illumination unit 5) by controlling the drive circuit 17. Specifically, the control device 6 can control the timing at which the solid light source 15 starts to turn on, the period during which the solid light source 15 is lit, the amount of illumination light emitted from the solid light source 15, the timing at which the solid light source is turned off, and the like.

  The cylindrical lens 16 is disposed at a position where the illumination light emitted from the solid light source 15 enters. The cylindrical lens 16 extends in the arrangement direction (longitudinal direction) of the plurality of solid light sources 15 on a surface orthogonal to the emission direction of the illumination light from the solid light sources 15, and is provided in common by the plurality of solid light sources 15. . That is, illumination light emitted from two or more solid light sources 15 among the plurality of solid light sources 15 enters the same cylindrical lens 16.

  As shown in FIG. 5B, the cylindrical lens 16 has power in a plane orthogonal to the arrangement direction of the plurality of solid light sources 15. The cylindrical lens 16 is provided so that the illumination light is condensed on the real image plane J in the short direction. As described above, the cylindrical lens 16 is provided so as to limit the spread of the illumination light in the normal direction of the real image plane J when the illumination light emitted from the plurality of solid light sources 15 propagates along the real image plane J. It has been. In addition, the illumination part 5 restrict | limits the spreading | diffusion of the illumination light in the normal line direction of the real image surface J when illumination light propagates along the real image surface J using the condensing effect | action of the retrotransmissive material 4. FIG. As such, it may be provided.

  Next, an illumination method by the illumination unit 5 will be described with reference to FIG. 6A shows a state of illumination in the longitudinal direction of the illuminating unit 5, and substantially corresponds to a diagram viewed from the front of the real image plane J (the shooting direction Dp of the camera 3 in FIG. 3A). FIG. 6B shows a state of illumination in the short direction of the illumination unit 5 and corresponds to a view seen from the side of the real image plane J.

  The light emitted from the solid light source 15 of the illuminating unit 5 converges toward a position (image formation position of the light source image) symmetrical to the solid light source 15 with respect to the retrotransmissive material 4, and from the image formation position of the light source image. As the distance from the retrotransmissive material 4 increases, it spreads and spreads. Here, due to the condensing action of the cylindrical lens 16 and the retrotransmissive material 4, the spread of the illumination light before the imaging position of the light source image is limited. In this way, the illumination light is propagated along the real image plane J with the spread in the normal direction of the real image plane J being limited. In other words, the attachment position and orientation of the illumination unit 5 are set, for example, with respect to the retrotransmissive material 4 so that the spread of illumination light in the normal direction of the real image plane J is limited.

  Here, as shown in FIG. 6B, a state is assumed in which an indicator Fin such as a pointer or a finger is arranged so that a user such as an observer K touches the real image plane J (real image M). In this state, the illumination light diffuses on the surface of the indicator Fin, and the camera 3 captures the indicator Fin illuminated by the illumination light. As described above, since the spread of the illumination light in the direction orthogonal to the real image plane J is limited, the shift between the position pointed by the user and the position of the indicator Fin in the portion illuminated by the illumination light is reduced. . The control device 6 shown in FIG. 1 detects the position of the indicator Fin in the portion illuminated by the illumination light based on such a photographed image, and accepts an input associated with the detected position.

  For example, depending on the structure of the retrotransmissive material 4, part of the illumination light directed from the illumination unit 5 toward the retrotransmissive material 4 does not reflect on the reflective surface 9 of the retrotransmissive material 4 and passes straight upward. There is a case. In the present embodiment, a light shielding member 18 for limiting the illumination area A1 is provided so that the illumination light does not illuminate the indicator Fin at a position that is a predetermined distance or more away from the real image plane J.

  Next, a mechanism for detecting the position of the indicator Fin will be described with reference to FIGS. FIG. 7 is a block diagram showing a functional configuration of the control device 6. FIG. 8 is a timing chart showing the timing of control by the control device 6.

  As shown in FIG. 7, the control device 6 includes a control unit 51 and a detection unit 52. The control unit 51 controls one or both of the photographing timing and the illumination state so as to associate the photographing timing of the camera 3 and the illumination state of the illumination unit 5. The detection unit 52 detects the indicator Fin illuminated by the illumination unit 5 based on the captured image captured by the camera 3. Here, description will be made assuming that the indicator Fin is a fingertip.

  The control unit 51 includes a synchronization separation unit 53, a timing generation unit 54, and an illumination control unit 55.

  The synchronization separation unit 53 acquires photographed image data indicating a photographed image from the camera 3. The synchronization separation unit 53 separates the synchronization signal (vertical synchronization signal, horizontal synchronization signal) from the captured image data, and outputs the separated synchronization signal to the timing generation unit 54.

  The timing generation unit 54 generates a timing signal (see “illumination” in FIG. 8) that defines the lighting start and lighting end (extinguishing) timings of the illumination unit 5 based on the synchronization signal from the synchronization separation unit 53. To do. As shown in FIG. 8, in the present embodiment, the vertical synchronization signal switches between high and low for each video signal corresponding to one frame of a captured image. The timing generation unit generates a timing signal that becomes low in an arbitrary period of one frame and becomes high in the period of one frame following the frame. The timing generation unit 54 outputs the generated timing signal to each of the illumination unit control unit 55 and the detection unit 52.

  The illumination unit control unit 55 switches the solid state light source 15 of the illumination unit 5 between lighting (first lighting state) and turning off (second lighting state) at a predetermined timing in accordance with the timing signal from the timing generation unit 54. That is, the illumination control unit 55 turns off the illumination unit 5 during a period when the timing signal is low, and turns on the illumination unit 5 during a period when the timing signal is high.

  The detection unit 52 includes a frame memory 56, a difference detection unit 57 (comparison unit), a fingertip extraction unit 58, a tip detection unit 59, a coordinate conversion unit 60, and a touch detection unit 61.

  The frame memory 56 stores captured image data from the camera 3. The frame memory 56 stores captured image data captured by the camera 3 in a predetermined illumination state based on the timing signal from the timing generation unit 54. As described above, the timing signal defines the timing of lighting start and lighting end (extinguishment), and the frame memory 56 uses the timing signal so that the captured image data of each frame and the illumination state at the time of shooting Can be associated. The frame memory 56 includes first photographed image data photographed by the camera 3 in the first illumination state in which the illumination unit 5 is turned on, and the illumination unit 5 in which the illumination unit 5 is turned off one time before or after this photographing. The second captured image data captured by the camera 3 in the illumination state 2 is output to the difference detection unit 57.

  The difference detection unit 57 compares the first captured image data from the frame memory 56 with the second captured image data. Here, the brightness of the captured image in the second illumination state in which the illumination unit 5 is turned off is almost the brightness of the ambient light, and the image is captured in the first illumination state in which the illumination unit 5 is turned on. The brightness of the fingertip illuminated in the second illumination state is added to the image. In other words, for each pixel of the photographed image, when the second photographed image data is subtracted from the first photographed image data, the background illuminated mainly by ambient light can be removed. The difference detection unit 57 outputs difference data indicating the difference between the first captured image data and the second captured image data to the fingertip extraction unit 58.

  By the way, the background may change due to, for example, the movement of the observer K between when the first image data is captured and when the second image data is captured. Therefore, for example, by increasing the output of the illumination unit 5 and making the illumination light on the real image plane J sufficiently brighter than the ambient light, the differential output of the fingertip can be made larger than the differential output of the background. In addition, the illumination unit 5 is configured to illuminate with infrared emission, and the camera 3 is provided with an infrared filter that selectively transmits infrared light of this wavelength, so that the background can be separated more easily. In addition, even if part of the light from the illumination unit 5 reaches the eyes of the observer K, it can be made invisible.

  Based on the difference data from the difference detection unit 57, the fingertip extraction unit 58 extracts an area occupied by the fingertip from the captured image showing the fingertip illuminated by the illumination light. The fingertip extraction unit 58 compares the brightness of the first captured image data and the second captured image data by comparing a preset threshold value (threshold) with the brightness indicated by the difference data for each pixel of the captured image. It is assumed that pixels on the photographed image whose length is different from the threshold value are extracted, and such a set of pixels on the photographed image is an area where the fingertip is reflected. The fingertip extraction unit 58 outputs region data indicating the extracted region (extraction region) to each of the tip detection unit 59 and the touch detection unit 61.

  The tip detection unit 59 detects a characteristic position of the extraction region extracted as the fingertip based on the region data from the fingertip extraction unit 58. The characteristic position may be, for example, the barycentric position of the extraction area or the tip position of the fingertip, and is appropriately set from positions that can be associated with the extraction area. For example, the tip detection unit 59 may determine which part of the fingertip the extraction region is based on pattern recognition or the like, and may detect the tip position of the fingertip based on the determination result. The tip detection unit 59 uses the detected characteristic position as a touch position, and outputs position data indicating the touch position to the coordinate conversion unit 60.

  By the way, the touch position indicated by the position data is information indicating the position of the pixel on the captured image, for example. When the captured image is captured outside the range of the real image M corresponding to the image V, the touch position is the touch position on the image V corresponding to the actual touch position on the real image M (on the image display surface 2a). In general, the pixel position is different. Further, even when the image format (pixel arrangement) of the captured image is different from the image format of the image V (image display surface 2a), the touch position indicated by the position data by the captured image is the touch position (image display) on the image V. The position of the pixel on the surface 2a is generally different.

  Therefore, the coordinate conversion unit 60 calculates coordinates corresponding to the touch position on the image V based on the position data from the tip detection unit 59. Here, since the formation range of the real image M occupying the shooting range of the camera 3 is determined by the arrangement of the camera 3, the arrangement of the FPD 2, and the like, the range of the real image M that appears in the shot image can be known information. The image format of the image captured by the camera 3 and the image format of the image V displayed by the FPD 2 are known information determined by the model of the camera 3, the model of the FPD 2, and the like. The coordinate conversion unit 60 converts the touch position indicated by the position data into coordinates on the image V displayed by the FPD 2 (pixel positions on the image display surface 2a) using known information as described above. The coordinate conversion unit 60 outputs information indicating the coordinates as input information from the user to an input processing unit (not shown). This input processing unit executes, for example, processing related to application operation or the like according to the input information.

  The touch detection unit 61 detects whether or not the fingertip has touched the real image plane J based on the area data from the fingertip extraction unit 58. For example, the touch detection unit 61 determines that a state in which no area data is output from the fingertip extraction unit 58 is a state in which the fingertip is not touching the real image plane J. The touch detection unit 61 outputs touch presence / absence information indicating the determination result to the input processing unit as described above. This touch presence / absence information is used for, for example, processing when the display device 1 is returned from the power saving mode.

  Next, an example of an input direction with respect to the display device 1 will be described with reference to FIGS. 9 and 10. FIG. 9 is a conceptual diagram illustrating an example of the input method, and FIG. 10 is a conceptual diagram illustrating another example of the input method.

  In the example shown in FIG. 9, the user can execute a process associated with the button PB by pressing the button PB displayed as the real image M. In this example, the control device 6 causes the FPD 2 to display an image V including a graphic indicating the button PB. As described with reference to FIG. 7, the control device 6 acquires the touch position on the image V corresponding to the touch position (detection position) on the real image M touched by the user, and the touch position on the image V and the button Based on the relationship with the BP, it is determined whether or not the user has touched the button. When it is determined that the user has performed an operation of touching the button BP, the control device 6 executes a process associated with the button BP. This process may be a process for causing a device external to the display device 1 to execute the process.

  The display device 1 displays the image V so that the button BP is arranged in a predetermined region of the image V, for example, the lower part or corner of the image V, and the camera 3 corresponds to a predetermined region of the real image plane J. Only the area to be captured may be photographed. Thus, the shooting range of the camera 3 may be a part of the real image plane J.

  In the example shown in FIG. 10, the user can input characters and figures by moving the fingertip on the real image plane J. In this example, the control device 6 acquires a touch position on the image V at a predetermined time interval, and receives a user input based on a change in the touch position over time. For example, the control device 6 causes the FPD 2 to display an image showing characters and figures drawn by the user with the fingertip.

  The display device 1 configured as described above illuminates the illumination area A1 where the real image M is formed by the retrotransmissive material 4 with the illumination unit 5 and shoots the illumination area A1 with the camera 3, so that the real image M is displayed. The indicator Fin that is about to be touched can be detected. The display device 1 can easily detect an input from the user by performing an operation in which the user touches the aerial image, and can satisfactorily perform an input on the screen displayed in the space.

  Further, the display device 1 controls the control device 6 to associate the photographing timing of the camera with the illumination state of the illumination unit, and compares the photographed images with different illumination states, so that the indicator Fin shown in the photographed image is displayed. It is possible to remove backgrounds other than those, and the indicator Fin can be detected with high accuracy.

  Moreover, the display apparatus 1 makes the state in which the solid light source 15 of the illumination unit 5 is lit as the first illumination state, and the state in which the solid light source 15 of the illumination unit 5 is unlit as the second illumination state. Yes. Therefore, the difference between the brightness of the indicator Fin reflected in the photographed image taken in the first illumination state and the brightness of the indicator Fin reflected in the photographed image photographed in the second illumination state is clarified. The indicator Fin can be detected with high accuracy.

  In the display device 1, the illumination unit 5 is disposed on the same side as the image display surface 2 a with respect to the element surface S. ) Is reduced, and such members are unlikely to obstruct the observation of the real image M. As a result, the display device 1 can perform a good-looking display.

  Further, since the illumination light propagates along the real image plane J, the display device 1 can accurately associate the position of the indicator Fin illuminated by the illumination light with the position on the real image plane J. In addition, since the spread of the illumination light is limited in the normal direction of the real image plane J, the display device 1 can accurately detect the indicator Fin illuminated by the illumination light.

  Further, since the shooting range on the real image plane J of the camera 3 includes the entire range in which the real image M can be formed on the real image plane J, the display device 1 is arranged at any position on the real image plane J. The indicated indicator Fin can be detected, and various inputs can be handled. Further, since the display device 1 displays the image V generated by the FPD 2 as a real image M, the device cost can be reduced.

  As described above, the display device 1 makes it possible to satisfactorily perform input on the screen displayed in the space. In addition, the display device 1 can be configured such that only the camera 3 exists above the retrotransmissive material 4, and even when the real image M is not displayed, the display device does not deteriorate in appearance and does not obstruct traffic. Damage is also difficult to occur.

  Further, when an unspecified number of users operate the display device 1 in applications such as signage and medical fields, the display device 1 is different from a device that performs input by touching an actual button (real image). Since the input can be performed by touching the surface J), for example, it can be used in a sanitary manner.

  In the present embodiment, the control device 6 controls the illumination unit 5 in synchronization with the camera 3, but may control the camera 3 in synchronization with the illumination unit 5. For example, the illumination unit 5 is configured to repeatedly turn on and off at predetermined intervals, and the control device 6 controls the shooting timing of the camera 3 based on a timing signal indicating the lighting timing of the illumination unit 5. Also good. In this case, the control device 6 may cause the camera 3 to perform shooting based on the timing signal when the illumination unit 5 is turned on and off.

  Next, a modified example of the retrotransmissive material 4 will be described.

  FIG. 11 is a perspective view showing Modification Example 1 of the retrotransmissive material 4. A retrotransmissive material 11 shown in FIG. 11 is a combination of a plurality of plate materials 12 such as metal or glass whose both surfaces or one surface is a reflective surface, which are orthogonal to each other. When only one surface of the plate material 12 is a reflection surface, the plate material 7 is arranged so that a pair of reflection surfaces that intersect with each other face the image generation surface of the image generation apparatus. Air may exist in the space between the adjacent board | plate materials 12, and the resin material etc. with high transparency may be filled.

  FIG. 12 is a perspective view showing Modification Example 2 of the retrotransmissive material 4. The retrotransmissive material 13 shown in FIG. 12 makes two sets of a plurality of glass materials 14 arranged such that one surface in the longitudinal direction of the prismatic glass material 14 is a mirror surface and the mirror surfaces are directed in the same direction. Two sets are stacked so as to be orthogonal to each other.

[Second Embodiment]
Next, a second embodiment will be described with reference to FIG. The display device 21 of the present embodiment is different from the first embodiment in the configuration of the image generation device, and the other configurations are the same as those in the first embodiment.

  FIG. 13 is a perspective view schematically showing a schematic configuration of the display device 21 of the present embodiment. In FIG. 13, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof may be simplified or omitted.

  The display device 21 of this embodiment includes a projection system 22 (image generation device), a camera 3, and a retrotransmissive material 4 (imaging element). The projection system 22 includes a transmissive screen 23 and a projector 24 that projects an image V onto the screen 23.

  In the present embodiment, the screen 23 is disposed in place of the FPD 2 at the position of the FPD 2 of the first embodiment. That is, the screen 23 is arranged so that the image display surface 23 a of the screen 23 is inclined with respect to the element surface S of the retrotransmissive material 4.

  The display device 21 configured as described above makes it possible to satisfactorily perform input on the screen displayed in the space. Further, since the display device 21 generates the image V by the projection system 22, the degree of freedom of the screen size is increased. The display device 21 can also display a bright real image M, for example, by increasing the screen gain.

[Third Embodiment]
Next, a third embodiment will be described with reference to FIG. The display device 30 of the present embodiment is different from the first embodiment in that it includes a plurality of processing units that form a real image and acquire a captured image.

  FIG. 14 is a perspective view schematically showing a schematic configuration of the display device 30 of the present embodiment. In the present embodiment, the retrotransmissive material 4 has a large number of openings 8 as shown in FIGS. 3A and 3B, but FIG. One of them is typically enlarged and schematically shown.

  The display device 30 illustrated in FIG. 14 includes a first processing unit 31, a second processing unit 32, a third processing unit 33, and a fourth processing unit 34 as a plurality of processing units. These processing units have the same configuration, but the positional relationship of the retrotransmissive material 4 with respect to the reflection surface 9 is different from each other.

  The first reflecting surface 9a and the second reflecting surface 9b of the retrotransmissive material 4 are orthogonal to each other in the first quadrant direction in the first quadrant with the center of the opening 8 as the origin. The 1st process part 31 is arrange | positioned so that a real image may be formed using the 1st reflective surface 9a and the 2nd reflective surface 9b. That is, the image generation device 35 of the first processing unit 31 is arranged with the image generation surface 35a facing the first reflection surface 9a and the second reflection surface 9b when viewed from the normal direction of the element surface S. The camera 36 of the first processing unit 31 is arranged so as to be able to photograph the real image plane J1 with respect to the image generation device 35.

  Note that the image generation apparatus 35 may be the FPD 2 described in the first embodiment, or the projection system 22 described in the second embodiment. The same applies to the second to fourth processing units 32 to 34.

  The second reflection surface 9b and the third reflection surface 9c are orthogonal to each other in the fourth quadrant direction (the rotation angle is greater than 270 ° and less than 360 °). The 2nd process part 32 is arrange | positioned so that a real image may be formed using the 2nd reflective surface 9b and the 3rd reflective surface 9c. The image generation device 37 of the second processing unit 32 is arranged with the image generation surface 37a facing the second reflection surface 9b and the third reflection surface 9c when viewed from the normal direction of the element surface S. The camera 38 of the second processing unit 32 is arranged so as to be able to photograph the real image plane J2 corresponding to the image generation device 37.

  The third reflecting surface 9c and the fourth reflecting surface 9d are orthogonal to each other in the third quadrant direction (the rotation angle is greater than 180 ° and smaller than 270 °). The 3rd process part 33 is arrange | positioned so that a real image may be formed using the 3rd reflective surface 9c and the 4th reflective surface 9d. The image generation device 39 of the third processing unit 33 is disposed with the image generation surface 39a facing the third reflection surface 9c and the fourth reflection surface 9d when viewed from the normal direction of the element surface S. The camera 40 of the third processing unit 33 is arranged so that the real image plane J3 corresponding to the image generation device 41 can be photographed.

  The fourth reflecting surface 9d and the first reflecting surface 9a are orthogonal to each other in the second quadrant direction (the rotation angle is greater than 90 ° and smaller than 180 °). The 4th process part 34 is arrange | positioned so that a real image may be formed using the 4th reflective surface 9d and the 1st reflective surface 9a. The image generation device 41 of the fourth processing unit 34 is arranged with the image generation surface 41a facing the fourth reflection surface 9d and the first reflection surface 9a when viewed from the normal direction of the element surface S. The camera 42 of the fourth processing unit 34 is arranged so as to be able to photograph the real image plane J4 corresponding to the image generating device 42.

  In the present embodiment, in the display device 30, the plurality of processing units 31 to 34 form images on the respective real image planes J1 to J4, and each of the plurality of images can be observed from a viewpoint corresponding to each processing unit. . These four viewpoints are arranged, for example, at the corners of the rectangular retrotransmissive material 4. In the display device 30 according to the present embodiment, auxiliary members 44 having a rectangular outer periphery are provided outside the retrotransmissive material 4, and corners (viewpoints) of the retrotransmissive material 4 are provided on each side of the outer periphery of the auxiliary member 44. Is arranged.

  The display device 30 configured as described above can handle, for example, four users, and is highly convenient. In addition, such a display device 30 uses, for example, a plurality of display devices 1 having a single image generation device and uses a common recursive transmission material 4 as compared with a configuration corresponding to a plurality of users. Can do.

  In the present embodiment, the number of processing units is four, but may be two or three. When the number of processing units is 2 or 3, the position of the processing unit can be appropriately selected from the first processing unit 31 to the fourth processing unit 34 described above.

  The technical scope of the present invention is not limited to the above embodiment. The requirements described in the above embodiments can be combined as appropriate. In addition, at least one of the requirements described in the above embodiment may be omitted.

  In the above-described embodiment, the state where the solid light source 15 of the illumination unit 5 is turned on is the first illumination state, and the state where the solid light source 15 of the illumination unit 5 is turned off is the second illumination state. However, the lighting state is not limited to this. For example, the second illumination state is a state in which the solid light source 15 of the illumination unit 5 is lit, and is darker than the first illumination state (a state in which the amount of light from the solid light source 15 is small). There may be. In this case, even if a part of the illumination light from the illumination unit is seen by the observer K, the difference in intensity of the illumination light is smaller than that of blinking, so that the uncomfortable feeling can be reduced.

DESCRIPTION OF SYMBOLS 1 ... Display apparatus, 2 ... Flat panel display (FPD, image generation apparatus), 2a ... Image display surface (image generation surface), 3 ... Camera, 3a ... Shooting range, 4. .. Retroreflective material (imaging element), 5... Illumination unit, 6... Control device (control unit, detection unit), 11, 13. ... Solid light source, 21 ... Display device, 22 ... Projection system (image generation device), 23 ... Screen, 23a ... Image display surface (image generation surface), 24 ... Projector, 51 ... Control unit, 52 ... Detection unit

Claims (9)

An image generating device for generating an image;
The image generating apparatus has an element surface that is inclined with respect to the image generation plane and on which light from the image is incident. An image element;
An illumination unit that illuminates an illumination area including the real image plane;
A camera for photographing the illumination area;
A display device comprising: a detection unit that detects an indicator that indicates the real image plane based on a captured image captured by the camera. The illumination unit has a first illumination state and a second illumination state,
A control unit that controls one or both of the timing of acquiring the captured image and the illumination state so as to associate the timing of acquiring the captured image of the camera and the illumination state of the illumination unit;
The detection unit detects the indicator by comparing a captured image captured by the camera in the first illumination state with a captured image captured by the camera in the second illumination state. The display device described in 1. The display device according to claim 1, wherein the illumination unit is disposed on the same side as the image generation surface with respect to the element surface, and illuminates the illumination region via the imaging element. The display device according to claim 1, wherein the illumination unit is provided so that illumination light propagates along the real image plane. The illumination unit includes a plurality of solid light sources arranged in a predetermined direction, emits spot-like illumination light that is emitted from the plurality of solid light sources and has the predetermined direction as a longitudinal direction, and is normal to the real image plane The display device according to claim 4, wherein the display device is provided so as to limit a spread of the illumination light in a direction. The display device according to any one of claims 1 to 5, wherein an imaging range on the real image plane of the camera includes an entire range in which the imaging element can form the image on the real image plane. A plurality of processing units including the image generation device and the camera paired with the image generation device;
The imaging element includes a first set of reflective surfaces that face each other in the first direction and a second set of reflective surfaces that intersect each other in the second direction;
The image generation device of the first processing unit is arranged with the image generation surface facing the first set of reflection surfaces,
The display device according to claim 1, wherein the image generation device of the second processing unit is arranged with the image generation surface facing the second set of reflection surfaces. The display device according to claim 1, wherein the image generation device includes a flat panel display having the image generation surface as a screen. The image generation device includes:
A screen disposed on the image generation surface;
The display apparatus as described in any one of Claims 1-7 provided with the projector which projects the said image on the said screen.

Images