KR102309397B1 - Cross-render multiview cameras, systems and methods - Google Patents

Abstract

A cross-render multi-view camera provides a multi-view image of a scene using a composite image generated from a difference map of the scene. The cross-render multiview camera includes a plurality of cameras along a first axis and is configured to capture a plurality of images of the scene. The cross-render multiview camera further comprises an image synthesizer configured to generate a composite image from the difference map determined from the plurality of images, wherein the composite image is a viewpoint corresponding to a position of the virtual camera on a second axis displaced from the first axis. Represents a view of the scene from. The cross-render multiview system further includes a multiview display configured to display the multiview image. A cross-render multiview imaging method includes capturing a plurality of images of a scene and generating a composite image using a difference map.

Description

Cross-render multiview cameras, systems and methods

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority from U.S. Provisional Patent Application No. 62/608,551, filed December 20, 2017, the contents of which are incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

N/A

BACKGROUND Electronic displays are a very common medium for conveying information to users of a wide variety of devices and products. The most commonly used electronic displays are a cathode ray tube (CRT), a plasma display panel (PDP), a liquid crystal display (LCD), an electroluminescent (EL) display, and an organic light emitting diode display. Organic light emitting diode (OLED) and active matrix OLED (AMOLED) displays, electrophoretic (EP) displays and various displays using electromechanical or electrofluidic light modulation ( digital micromirror devices, electrowetting displays, etc.). In general, electronic displays can be classified as active displays (ie, displays that emit light) or passive displays (ie, displays that modulate light provided by another source). The most obvious examples of active displays are CRTs, PDPs and OLED/AMOLEDs. The displays that are generally classified as passive when considering the emitted light are LCD and EP displays. Passive displays often exhibit attractive performance characteristics, including, but not limited to, inherently low power consumption, but may have somewhat limited use in many practical applications lacking the ability to emit light.

Image capture, and particularly three-dimensional (3D) image capture, is the process of converting captured images (eg, generally two-dimensional images) into 3D images for display on a 3D display or multi-view display. This generally involves substantial image processing of the captured images. Image processing may include, but is not limited to, depth estimation, image interpolation, image reconstruction, or other complex processes that can create significant time delays from the moment the images are captured to the moment the images are displayed.

BRIEF DESCRIPTION OF THE DRAWINGS Various features of examples and embodiments in accordance with the principles described herein may be more readily understood by reference to the following detailed description taken in connection with the accompanying drawings in which like reference numerals indicate like structural elements.
1A illustrates a perspective view of a multiview display as an example in accordance with an embodiment consistent with the principles described herein;
1B shows a graphical representation of angular components of a light beam having a particular principal angular direction corresponding to the viewing direction of a multi-view display as an example according to an embodiment consistent with the principles described herein;
2A shows a diagram of a cross-render multi-view camera as an example in accordance with an embodiment consistent with the principles described herein.
2B shows a perspective view of a cross-render multi-view camera as an example in accordance with an embodiment consistent with the principles described herein.
3A illustrates a graphical representation of images associated with a cross-render multi-view camera as an example according to an embodiment consistent with the principles described herein.
3B illustrates a graphical representation of images associated with a cross-render multi-view camera as another example according to an embodiment consistent with the principles described herein.
4 shows a block diagram of a cross-render multiview system 200 as an example in accordance with an embodiment consistent with the principles described herein.
5A illustrates a cross-sectional view of a multi-view display as an example in accordance with an embodiment consistent with the principles described herein.
5B shows a top view of a multiview display as an example in accordance with an embodiment consistent with the principles described herein.
5C illustrates a perspective view of a multiview display as an example in accordance with an embodiment consistent with the principles described herein.
6 illustrates a cross-sectional view of a multi-view display including a wide-angle backlight as an example in accordance with an embodiment consistent with the principles described herein.
7 shows a flowchart of a cross-render multiview imaging method as an example in accordance with an embodiment consistent with the principles described herein.
Some examples and embodiments may have other features in addition to or included in place of the features shown in the figures described above. These and other features are described below with reference to the drawings above.

Embodiments and examples in accordance with the principles described herein provide multiview or 'holographic' imaging that may correspond to or be used with a multiview display. In particular, according to various embodiments of the principles described herein, multiview imaging of a scene may be provided by a plurality of cameras arranged along a first axis. The plurality of cameras are configured to capture a plurality of images of the scene. The image is then synthesized to generate a synthesized image representing a view of the scene from a perspective corresponding to the position of the virtual camera on a second axis displaced from the first axis. synthesis) is used. According to various embodiments, the composite image is generated by image synthesis from a parallax or a depth map of a scene. According to various embodiments, thereafter, a multi-view image including a composite image may be provided and displayed. The multi-view image may further include an image of a plurality of images. The one or more composite images and one or more images of the plurality of images may be viewed together as a multi-view image on the multi-view display. Also, viewing a multi-view image on a multi-view display may allow the user to perceive elements in the multi-view image of a scene at different apparent depths within the physical environment when viewed on the multi-view display, including the cameras viewpoint views of a scene that do not exist in the plurality of images captured by As such, according to some embodiments, a cross-render multiview camera according to an embodiment of the principles described herein, when viewed on a multiview display, is only available to a viewer with a plurality of cameras. It can create multi-view images that provide a 'more complete' three-dimensional (3D) viewing experience than possible.

As used herein, a 'two-dimensional display' or a '2D display' refers to a view of a displayed image that is substantially the same regardless of the direction in which the displayed image is viewed on the 2D display (i.e., within a defined viewing angle or range of the 2D display). It is defined as a display configured to provide LCDs found in smart phones and computer monitors are examples of 2D displays. In contrast, herein, a 'multiview display' is defined as a display or display system configured to provide different views of a multiview image in different viewing directions or from different viewing directions. In particular, different views may represent different perspective views of an object or scene in a multiview image. In some examples, a multiview display may be referred to as a three-dimensional (3D) display, for example, when viewing two different views of a multiview image simultaneously provides the perception that viewing a three-dimensional (3D) image. . Multiview displays and uses of multiview systems applicable to the capture and display of multiview images described herein include mobile phones (eg, smart phones), watches, tablet computers, mobile computers ( eg, laptop computers), personal computers and computer monitors, vehicle display consoles, camera displays, and various other removable as well as substantially non-removable display applications and appliances. .

1A shows a perspective view of a multiview display 10 according to an example consistent with the principles described herein. As shown, the multi-view display 10 includes a screen 12 that is shown for viewing the multi-view image. The multi-view display 10 provides different views 14 of the multi-view image in different viewing directions 16 with respect to the screen 12 . The viewing directions 16 are shown as arrows extending from the screen 12 in several different principal angular directions, the different views 14 being shaded at the distal end of the arrows representing the viewing directions 16 . Shown as indicated polygonal boxes, only four views 14 and view directions 16 are shown by way of example and not limitation. Although the different views 14 are shown in FIG. 1A as being on the screen, when the multiview image is displayed on the multiview display 10 the views 14 are actually on or near the screen 12 . Note that it appears The depiction of views 14 above screen 12 is for illustrative simplicity only, and to indicate looking at multiview display 10 from each view direction 16 corresponding to a particular view 14 . am. Further, the views 14 of the multiview display 10 and their corresponding view directions 16 are generally aligned or arranged in a particular arrangement depending upon the implementation of the multiview display 10 . For example, as will be described below, views 14 and corresponding view directions 16 can have a rectangular arrangement, a square arrangement, a circular arrangement, a hexagonal arrangement, etc. dictated by the particular multiview display implementation.

According to the definition herein, a light beam having a viewing direction or, equivalently, a direction corresponding to the viewing direction of a multi-view display generally has a principal angular direction given by the angular components {θ, φ}. The angle component θ is referred to herein as the 'elevation component' or 'elevation angle' of the light beam. The angular component φ is referred to as the 'azimuth component' or 'azimuth angle' of the light beam. By definition, elevation angle θ is the angle in a vertical plane (eg perpendicular to the plane of the screen of a multiview display), and azimuth angle φ is the angle in a horizontal plane (eg, the screen of a multiview display). is the angle parallel to the plane of

1B shows a graphical representation of the angular components {θ, φ} of a light beam 20 with a particular principal angular direction corresponding to the viewing direction of a multiview display according to an example of the principles described herein. Also, by definition herein, the light beam 20 is emitted or diverged from a specific point. That is, by definition, the light beam 20 has a central ray associated with a particular point of origin within the multi-view display. 1B also shows the origin O of the light beam (or direction of view).

In this specification, 'multiview' as used in the terms 'multiview image' and 'multiview display' means the angular disparity between views of a plurality of views. ) or are defined as a plurality of views representing different perspectives. Also, by definition, the term 'multiview' explicitly includes three or more different views (ie a minimum of three views, typically four or more views). Thus, 'multiview' as used herein is clearly distinct from stereoscopic views, which include only two different views, for example to represent a scene. However, by definition herein, multiview images and multiview displays contain three or more views, but by selecting only two of the views to be viewed simultaneously (eg, one view per eye). Note that multiview images may be viewed as stereoscopic pair of images (eg, on a multiview display).

As used herein, a 'multiview pixel' is a set or group of sub-pixels representing 'view' pixels of each view of a plurality of different views of a multiview display ( light valves). In particular, a multi-view pixel may have a separate sub-pixel corresponding to or representing a view pixel of each of the different views of the multi-view image. Further, according to the definition herein, sub-pixels of a multi-view pixel are so-called 'directional pixels' in that each of the sub-pixels relates to a given direction of view of a corresponding one of the different views. . Further, according to various examples and embodiments, different view pixels represented by sub-pixels of a multi-view pixel may have equal or at least substantially similar positions or coordinates in each of the different views. For example, a first multiview pixel may have individual sub-pixels corresponding to view pixels located at { x ₁ , y ₁ } of each of the different views of the multiview image, and the second multiview pixel may have different views It may have individual sub-pixels corresponding to view pixels located at each { x ₂ , y _{2 }.}

In some embodiments, the number of sub-pixels in a multi-view pixel may be equal to the number of different views of the multi-view display. For example, a multiview pixel may provide 8, 16, 32 or 64 sub-pixels respectively associated with a multiview display having 8, 16, 32 or 64 different views. In another example, a multiview display may provide a 2x2 array of views (ie, 4 views), and a multiview pixel may include 4 sub-pixels (ie, one for each view). Additionally, each different sub-pixel may have an associated direction (ie, a principal angular direction of the light beam), eg, corresponding to a different one of the viewing directions corresponding to different views. Also, according to some embodiments, the number of multi-view pixels of the multi-view display may be substantially equal to the number of 'view' pixels (ie, pixels constituting the selected view) in the views of the multi-view display. For example, if a view includes 640x480 view pixels (ie, a view resolution of 640x480), the multiview display may have 307,200 multiview pixels. In another example, if the views include 100x100 pixels, the multiview display may include a total of 10,000 multiview pixels (ie, 100x100=10,000).

As used herein, a 'light guide' is defined as a structure that guides light therein using total internal reflection. In particular, the light guide may include a core that is substantially transparent at the operational wavelength of the light guide. The term 'light guide' generally refers to a dielectric optical waveguide that utilizes total internal reflection to guide light at the boundary between the dielectric material of the light guide and the material or medium surrounding the light guide. By definition, the condition for total internal reflection is that the refractive index of the light guide must be greater than the refractive index of the surrounding medium adjacent to the surface of the light guide material. In some embodiments, the light guide may include a coating in addition to or in lieu of the refractive index difference described above to further facilitate total internal reflection. For example, the coating may be a reflective coating. The light guide may be any of a variety of light guides including, but not limited to, one or both of plate or slab guides and strip guides.

Also herein, the term 'plate' when applied to a light guide as in a 'plate light guide', often referred to as a 'slab' guide, means a piece of -wise) or differentially planar layer or sheet. In particular, a plate light guide is defined as a light guide configured to guide light in two substantially orthogonal directions bounded by a top surface and a bottom surface (ie, opposing surfaces) of the light guide. Also, by definition herein, the top and bottom surfaces may be spaced apart from each other and substantially parallel to each other in at least a distinct sense. That is, within any distinctly small area of the plate light guide, the top and bottom surfaces are substantially parallel or co-planar.

In some embodiments, the plate light guide may be substantially planar (ie, confined to a plane), and thus the plate light guide is a planar light guide. In other embodiments, the plate light guide may be curved in one or two orthogonal dimensions. For example, a plate light guide may be curved in a single dimension to form a plate light guide of a cylindrical shape. However, any curvature has a radius of curvature large enough to ensure that total internal reflection is maintained within the plate light guide to guide the light.

As used herein, 'diffraction grating' is defined as a plurality of features (ie, diffractive features) arranged to provide diffraction of light incident on the diffraction grating. In some examples, the plurality of features may be arranged in a periodic or quasi-periodic manner. In other examples, the diffraction grating may be a mixed-period diffraction grating comprising a plurality of diffraction gratings, and each diffraction grating of the plurality of diffraction gratings may have a different periodic arrangement of features. Further, the diffraction grating may include a plurality of features (eg, a plurality of grooves or ridges in the material surface) arranged in a one-dimensional (1D) array. Alternatively, the diffraction grating may include a two-dimensional (2D) array of features or an array of features defined in two dimensions. For example, the diffraction grating may be a 2D array of bumps on the material surface or holes in the material surface. In some examples, the diffraction grating may be substantially periodic in a first direction or in a first dimension, and may be substantially aperiodic (eg, constant, random, etc.) in a direction across or along the diffraction grating in another direction. can

As such, and by definition herein, a 'diffraction grating' is a structure that provides diffraction of light incident on the diffraction grating. When light is incident on the diffraction grating from the light guide, the diffraction or diffractive scattering provided is 'diffractive coupling in that the diffraction grating can couple out light from the light guide by diffraction. (diffractive coupling)' and may therefore be referred to as such. Also, the diffraction grating redirects or changes the angle of light by diffraction (ie, at a diffractive angle). In particular, as a result of diffraction, light leaving the diffraction grating (ie diffracted light) generally has a propagation direction different from that of light incident on the diffraction grating (ie incident light). In this specification, the change in the direction of propagation of light by diffraction is referred to as 'diffractive redirection'. Thus, a diffraction grating can be understood to be a structure comprising diffractive features that diffractively redirect light incident on the diffraction grating, and when light is incident from the light guide the diffraction grating also diffracts the light from the light guide. You can couple out as enemies.

Also, by the definition herein, the features of a diffraction grating are referred to as 'diffractive features', on a surface (i.e., 'surface' refers to the boundary between two materials), within a surface. and on the surface. The surface may be the surface of the plate light guide. The diffractive features may include any of a variety of structures that diffract light, including, but not limited to, one or more of grooves, ridges, holes, and protrusions, such structures comprising: in one or more of and on the surface. For example, the diffraction grating may include a plurality of parallel grooves in the material surface. In another example, the diffraction grating may include a plurality of parallel ridges rising from the material surface. The diffractive features (any of grooves, ridges, holes, protrusions, etc.) may be of a sinusoidal profile, a rectangular profile (eg, a binary diffraction grating), a triangular profile, and a sawtooth profile (eg, a blaze grating). It may have any of a variety of cross-sectional shapes or profiles that provide diffraction, including but not limited to one or more.

According to various examples described herein, a diffraction grating (eg, the diffraction grating of a diffractive multibeam element as described below) is configured to diffract light from a light guide (eg, plate light guide) as a light beam. It can be used to scatter or couple out. _{In particular, the diffraction angle θ m} provided by or of a locally periodic diffraction grating can be given by equation (1).

(1)

(One)

where λ is the wavelength of the light, m is the diffraction order, n is the refractive index of the light guide, d is the distance or spacing between features of the diffraction grating, and θ _i is the angle of incidence of the light on the diffraction grating. For simplicity, equation (1) assumes that the diffraction grating is adjacent to the surface of the light guide and that the refractive index of the material outside the light guide is 1 (ie, n _out = 1). In general, the diffraction order (m) is given as an integer (ie, m = ± 1, ± 2, ...). _{The diffraction angle θ m} of the light beam generated by the diffraction grating can be given by Equation (1). When the diffraction order m is 1 (ie, m = 1), a first-order diffraction, more specifically a first-order diffraction angle θ _m , is provided.

Further, according to some embodiments, the diffractive features of the diffraction grating may be curved and may also have a defined orientation (eg, tilted or rotated) with respect to the direction of propagation of light. For example, one or both of the curve of the diffractive features and the orientation of the diffractive features can be configured to control the direction of light coupled out by the diffraction grating. For example, the principal angular direction of the directional light may be a function of the angle of the diffractive feature at the point at which the light is incident on the diffraction grating with respect to the direction of propagation of the incident light.

As used herein, a 'multibeam element' is a structure or element of a backlight or display that generates light including a plurality of light beams. By definition, a 'diffractive' multibeam device is a multibeam device that generates a plurality of light beams by or using diffractive coupling. In particular, in some embodiments, a diffractive multibeam element may be optically coupled to a light guide of a backlight to provide a plurality of light beams by diffractively coupling out a portion of the guided light within the light guide. have. Further, as defined herein, a diffractive multi-beam element includes a plurality of diffraction gratings within a boundary or extent of the multi-beam element. According to the definition herein, the light beams of the plurality of light beams generated by the multi-beam element have different principal angular directions from each other. In particular, by definition, any one of the plurality of light beams has a predetermined principal angular direction different from that of the other of the plurality of light beams. According to various embodiments, the spacing or grating pitch of the diffractive features in the diffraction gratings of the diffractive multibeam element may be sub-wavelength (ie, less than the wavelength of the guided light).

Although a multibeam element having a plurality of diffraction gratings is used as an illustrative example in the following discussions, in some embodiments other components, such as at least one of a micro reflective element and a micro refractive element, are used in the multibeam element. can be For example, the micro-reflective element may include a triangular-shaped mirror, a ladder-shaped mirror, a pyramid-shaped mirror, a rectangular-shaped mirror, a hemispherical-shaped mirror, a concave mirror, and/or a convex mirror. In some embodiments, the microrefractive element may include a triangular refractive element, a ladder-shaped refractive element, a pyramid-shaped refractive element, a rectangular refractive element, a hemispherical refractive element, a concave refractive element, and/or a convex refractive element.

According to various embodiments, the plurality of light beams may represent a light field. For example, the plurality of light beams may be confined to a substantially conical spatial region or have a defined angular spread comprising different principal angular directions of the light beams within the plurality of light beams. Thus, a given angular spread of light beams may represent a light field as a combination (ie, a plurality of light beams).

According to various embodiments, different principal angular directions of the different light beams in the plurality of light beams are related to the size of the diffractive multi-beam element (eg, one or more of length, width, area, etc.) and the diffraction grating within the diffractive multi-beam element, according to various embodiments. is determined by properties including, but not limited to, the orientation and 'grating pitch' or diffractive feature spacing. As defined herein, in some embodiments, a diffractive multibeam element is an 'extended point light source', ie a plurality of distributions across the extent of the diffractive multibeam element. can be considered as point sources of Also, by definition herein, and as described above with respect to FIG. 1B , the light beam produced by the diffractive multibeam element has a principal angular direction given by the angular components {θ, φ}.

As used herein, a 'collimator' is defined as substantially any optical instrument or device configured to collimate light. For example, a collimator may include, but is not limited to, a collimating mirror or reflector, a collimating lens, a collimating diffraction grating, and various combinations thereof.

In this specification, a 'collimation factor' denoted by σ is defined as the degree to which light is collimated. In particular, by definition herein, the collimation coefficient defines the angular spread of light rays within a beam of collimated light. For example, the collimation coefficient (σ) may specify that most rays within a beam of collimated light are within a certain angular spread (e.g., +/- σ degrees with respect to the center or principal angular direction of the collimated light beam). have. According to some examples, the rays of the collimated light beam may have a Gaussian distribution in terms of angles, and the angular spread may be an angle determined by half the peak intensity of the collimated light beam.

As used herein, a 'light source' is defined as a source of light (eg, a device or device that emits light). For example, the light source may be a light emitting diode (LED) that emits light when activated. The light source is substantially any source of light or includes, but is not limited to, one or more of LEDs, lasers, OLEDs, polymer LEDs, plasma-based optical emitters, fluorescent lamps, incandescent lamps, and virtually any other source of light; It may be an optical emitter. The light generated by the light source may have a color (ie, may include a specific wavelength of light) or may include a specific wavelength of light (eg, white light). Also, in this specification, a 'plurality of light sources of different colors' is explicitly defined as a set or group of light sources, wherein at least one of the light sources is the color of light produced by at least one other light source of the plurality of light sources. or a color different from the wavelength, or equivalently a wavelength. For example, different colors may include primary colors (eg, red, green, blue). Further, as long as at least two of the plurality of light sources are of different color light sources (ie, the at least two light sources produce different colors of light), a 'plurality of light sources of different colors' is of the same or substantially similar color. It may include two or more light sources. Thus, by definition herein, a 'plurality of light sources of different colors' may include a first light source producing a first color of light and a second light source producing a second color of light, the second color being different from the first color.

In this specification, 'arrangement' or 'pattern' is defined as a relationship between elements defined by the relative positions of elements and the number of elements. More specifically, as used herein, 'array' or 'pattern' does not define the spacing between elements or the size of the sides of the array of elements. As defined herein, a 'square' arrangement has an equal number of elements (eg, a camera) in each of two substantially orthogonal directions (eg, the x-direction and the y-direction). s, views, etc.) is a rectilinear arrangement of elements. On the other hand, a 'rectangular' arrangement is defined as an arrangement surrounded by a straight line comprising a different number of elements in each of two orthogonal directions.

Herein, by definition, a spacing or separation between elements of an array is referred to as a 'baseline' or equivalently a 'baseline distance'. For example, the cameras in an array of cameras may be spaced apart from each other by a reference distance that defines a spacing or distance between individual cameras in the camera array.

In addition, according to the definition of this specification, the term 'broad-angle' as in 'broad-angle emitted light' means the cone angle of a multi-view image or a multi-view display's view. ) is defined as light with a cone angle greater than . In particular, in some embodiments, the wide angle emission light may have a cone angle greater than about 60 degrees. In other embodiments, the cone angle of the wide angle emission light may be greater than about 50 degrees, or greater than about 40 degrees. For example, the cone angle of the wide-angle emission light may be about 120 degrees. Alternatively, the wide angle emission light may have an angular range greater than ±45 degrees (eg, greater than ±45 degrees) relative to the normal direction of the display. In other embodiments, the angular range of the wide-angle emission light is greater than ±50 degrees (eg, greater than ±50 degrees), or greater than ±60 degrees (eg, greater than ±60 degrees), or ± may be greater than 65 degrees (eg, greater than ±65 degrees). For example, the angular range of the wide angle emission light may be greater than about 70 degrees (eg, greater than ±70 degrees) on either side of the normal direction of the display. As defined herein, a 'broad-angle backlight' is a backlight configured to provide wide-angle emission light.

In some embodiments, the cone angle of the wide angle emission light is equal to the viewing angle of an LCD computer monitor, LCD tablet, LCD television, or similar digital display device for a wide viewing angle (eg, about ± 40-65 degrees). can be defined as being about the same. In other embodiments, the wide angle emission light may also be diffuse light, substantially diffuse light, non-directional light (eg, lacking a specific or defined directionality), or a single or substantially uniform direction. can be characterized or described as light with

Embodiments consistent with the principles described herein include integrated circuits (ICs), very large scale integrated (VLSI) circuits, field programmable gate arrays (FPGAs), digital signal processors (DSPs), graphic processor units (GPUs), firmware, software (such as program modules or sets of instructions) and combinations of two or more thereof, but It may be implemented using various devices and circuits, but is not limited thereto. For example, the image processor or other elements described below may all be implemented as circuit elements within an ASIC or VLSI circuit. Implementations using ASIC or VLSI circuitry are examples of hardware-based circuit implementations.

In another example, embodiments of an image processor may be implemented in a computer-implemented operating environment or a software-based modeling environment (eg, MATLAB®, MathWorks, Inc., Natick, MA) running in a computer programming language (eg, Natick, MA). , C/C++) may be implemented as software (eg, stored in a memory and executed by a processor or graphics processor of a computer). One or more computer programs or software may constitute a computer-program mechanism, and a programming language may be compiled or interpreted for execution by a processor or graphics processor of a computer, eg, configurable or configured (as in this discussion). may be used interchangeably), there is.

In another example, some blocks, modules, or elements of an apparatus or system (eg, image processor, camera, etc.) described herein are implemented using real or physical circuitry (eg, IC or ASIC). may be implemented, and other blocks, modules or elements may be implemented in software or firmware. In particular, in accordance with the above definitions, some embodiments described herein may be implemented using substantially hardware-based circuit approaches or devices (eg, ICs, VLSIs, ASICs, FPGAs, DSPs, firmware, etc.). and other embodiments may be implemented as software or firmware using a computer processor or graphics processor to execute the software, or may be implemented as a combination of software or firmware and hardware-based circuitry, for example.

Also, as used herein, the terms "a" and "a" are intended to have their ordinary meanings in the patent art, ie, 'one or more'. For example, as used herein, 'camera' means one or more cameras, and thus 'the camera' means 'the camera(s)'. In addition, in the present specification, 'top', 'bottom', 'top', 'bottom', 'top', 'bottom', 'front', 'after', 'first', 'second', 'left' or reference to 'right' is not intended to be limiting herein. As used herein, unless expressly specified otherwise, the term 'about' when applied to a numerical value generally means within the tolerance of the equipment used to produce the numerical value, or ±10%, or ±5%, or ±1%. Also, the term 'substantially' as used herein means most, or almost all, or all, or an amount within the range of from about 51% to about 100%. Also, the examples herein are intended to be illustrative only, and are presented for purposes of discussion and not limitation.

According to some embodiments of the principles described herein, a cross-render multiview camera is provided. 2A shows a diagram of a cross-render multiview camera 100 as an example in accordance with an embodiment consistent with the principles described herein. 2B shows a perspective view of a cross-render multiview camera 100 as an example in accordance with an embodiment consistent with the principles described herein. The cross-render multiview camera 100 is configured to capture a plurality of images 104 of the scene 102 and then synthesize or create a composite image of the scene 102 . In particular, the cross-render multiview camera 100 captures a plurality of images 104 of the scene 102 representing different viewpoint views of the scene 102 , and then the different images represented by the plurality of images 104 . It may be configured to generate a composite image 106 representing a view of the scene 102 from a different viewpoint than the viewpoint views. Accordingly, according to various embodiments, the composite image 106 may represent a 'new' viewpoint view of the scene 102 .

As shown, the cross-render multi-view camera 100 includes a plurality of cameras 110 spaced apart from each other along a first axis. For example, as shown in FIG. 2B , the plurality of cameras 110 may be spaced apart from each other as a linear array in the x-direction. Accordingly, the first axis may include the x-axis. Note that although depicted as being on a common axis (ie, a linear array), in some embodiments sets of cameras 110 of a plurality of cameras may be arranged along several different axes (not shown).

The plurality of cameras 110 is configured to capture a plurality of images 104 of the scene 102 . In particular, each camera 110 of the plurality of cameras may be configured to capture a different one of the images 104 of the plurality of images. For example, the plurality of cameras may include two cameras 110 , and each camera 110 may be configured to capture a different one of the two images 104 of the plurality of images. For example, the two cameras 110 may represent a stereo pair of cameras or a 'stereo camera' for short. In other examples, the plurality of cameras is three cameras configured to capture three images 104 , or four cameras 110 configured to capture four images 104 , or five images ( 104) and five cameras configured to capture it. Also, different images 104 of the plurality of images may be different from the scene 102 because the cameras 110 are spaced apart from each other along a first axis (eg, the x-axis as shown). It shows different viewpoint views.

According to various embodiments, cameras 110 of a plurality of cameras may include virtually any camera or associated imaging or image capture device. In particular, cameras 110 may be digital cameras configured to capture digital images. For example, a digital camera may be a charge-coupled device (CCD) image sensor, a complementary metal-oxide semiconductor (CMOS) image sensor, or a back-side-illuminated CMOS (BSI) image sensor. -CMOS) sensors, such as, but not limited to, digital image sensors. Further, according to various embodiments, cameras 110 may be configured to capture one or both of still images (eg, photos) and moving images (eg, video). In some embodiments, cameras 110 capture amplitude or intensity and phase information in a plurality of images.

The cross-render multi-view camera 100 shown in FIGS. 2A and 2B further includes an image synthesizer 120 . The image synthesizer is configured to generate a composite image 106 of the scene 102 using a disparity map or a depth map of the scene 102 determined from the plurality of images. In particular, image synthesizer 120 may be configured to determine a difference map from images 104 of a plurality of images (eg, a pair of images) captured by the camera array. The image synthesizer 120 may then use one or more of the images 104 of the plurality of images and the determined difference map to generate the composite image 106 . According to various embodiments, any of a number of different approaches for determining a difference map (or equivalently a depth map) may be used. In some embodiments, image synthesizer 120 is further configured to provide hole-filling to one or both of difference map and composite image 106 . For example, image synthesizer 120 is described in the document Hamzah et al. in, "Literature Survey on Stereo Vision Disparity Map Algorithms," J. of Sensor, Vol. 2016, Article ID 8742920, each of which is incorporated herein by reference. ), documents (Jain et al., "Efficient Stereo-to-Multiview Synthesis," ICASSP 2011, pp. 889-892), or documents (Nquyen et al., "Multiview Synthesis Method and Display Devices with Spatial and Inter-View) Any of the methods described in Consistency, US 2016/0373715 A1) may be used.

According to various embodiments, the composite image 106 generated by the image synthesizer represents a view of the scene 102 from a viewpoint corresponding to the position of the virtual camera 110 ′ on a second axis displaced from the first axis. . For example, as shown in FIG. 2B , the cameras 110 of the plurality of cameras may be arranged in a linear manner along the x-axis and spaced apart from each other, and the virtual camera 110 ′ may be y from the plurality of cameras. direction can be displaced.

In some embodiments, the second axis is perpendicular to the first axis. For example, as shown in FIG. 2B , the second axis may be in the y direction (eg, the y-axis) if the first axis is in the x-direction. In other embodiments, the second axis may be parallel to but laterally displaced from the first axis. For example, both the first and second axes may be in the x direction, but the second axis may be displaced laterally in the y direction with respect to the first axis.

In some embodiments, image synthesizer 120 is configured to provide a plurality of composite images 106 using the difference map. In particular, each composite image 106 of the plurality of composite images may represent a view of the scene 102 from a different viewpoint of the scene 102 compared to other composite images 106 of the plurality of composite images. . For example, the plurality of composite images 106 may include two, three, four or more composite images 106 . Accordingly, the plurality of composite images 106 may represent views of the scene 102 corresponding to, for example, locations of a plurality of similar virtual cameras 110 ′. Also, in some examples, the plurality of virtual cameras 110 ′ may be located on one or more different axes corresponding to the second axis. In some embodiments, the number of composite images 106 may be equal to the number of images 104 captured by the plurality of cameras.

In some embodiments, the plurality of cameras 110 may include a pair of cameras 110a and 110b configured as a stereo camera. Also, the plurality of images 104 of the scene 102 captured by the stereo camera may include the stereo pair of images 104 of the scene 102 . In such embodiments, the image synthesizer 120 is configured to provide a plurality of composite images 106 representing views of the scene 102 from viewpoints corresponding to positions of the plurality of virtual cameras 110 ′. can be

In some embodiments, the first axis may be or may represent a horizontal axis, and the second axis may be or represent a vertical axis orthogonal to the horizontal axis. In these embodiments, the stereo pair of images 104 may be arranged in a horizontal direction corresponding to a horizontal axis, and a plurality of composite images including the pair of composite images 106 may be arranged in a vertical direction corresponding to the vertical axis. direction can be arranged.

3A shows a graphical representation of images associated with a cross-render multi-view camera 100 as an example according to an embodiment consistent with the principles described herein. In particular, the left side of FIG. 3A shows a stereo pair of images 104 of a scene 102 captured by a pair of cameras 110 serving as a stereo camera. As shown, the stereo pair of images 104 are arranged in a horizontal orientation, and thus may be referred to as being in a landscape orientation. The right side of FIG. 3A shows a stereo pair of composite images 106 generated by the image synthesizer 120 of the cross-render multiview camera 100 . As shown, the composite images 106 of the stereo pair's composite images 106 are arranged in a vertical orientation, and thus may be referred to as being in a portrait orientation. Arrows between the left and right stereo images represent the operation of image synthesizer 120 including determining a difference map and generating a stereo pair of composite images 106 . According to various embodiments, FIG. 3A may illustrate converting images 104 captured by a plurality of cameras in a horizontal direction into composite images 106 in a vertical direction. Conversely, although not explicitly shown, images 104 in portrait orientation (ie captured by vertically arranged cameras 110 ) are rendered by image synthesizer 120 in landscape orientation (ie, It is also possible to convert to composite images 106 (in a horizontal arrangement) or to provide composite images 106 .

FIG. 3B illustrates a graphical representation of images associated with a cross-render multi-view camera 100 , as another example according to an embodiment consistent with the principles described herein. In particular, the upper portion of FIG. 3B shows a stereo pair of images 104 of a scene 102 captured by a pair of cameras 110 serving as a stereo camera. The lower portion of FIG. 3B shows a stereo pair of composite images 106 generated by the image synthesizer 120 of the cross-render multiview camera 100 . In addition, the stereo pair of composite images 106 includes a pair of virtual cameras ( 110'). According to various embodiments, the stereo pair of images 104 captured by the cameras 110 are used to provide a so-called four-view (4V) multi-view image of the scene 102 . may be combined with the stereo pair of composite images 106 to provide views.

In some embodiments (not explicitly shown in FIGS. 2A and 2B ), the cross-render multiview camera 100 may further include a processing subsystem, a memory subsystem, a power subsystem and a networking subsystem. . The processing subsystem may include one or more devices configured to perform computational operations, such as, but not limited to, a microprocessor, graphics processor unit (GPU) or digital signal processor (DSP). can The memory subsystem may include one or more devices for storing one or both of data and instructions that may be used by the processing subsystem to provide and control operation of the cross-render multiview camera 100 . have. For example, the stored data and instructions may initiate capture of a plurality of images using the plurality of cameras 110 , implement the image synthesizer 120 , and the images 104 and composite image(s) ( 106) data and instructions configured to do one or more of displaying multi-view content on a display (eg, a multi-view display). For example, a memory subsystem may include one or more of, including, but not limited to, random access memory (RAM), read-only memory (ROM), and various types of flash memory. It can contain any type of memory.

In some embodiments, instructions stored in the memory subsystem and used by the processing subsystem include, but are not limited to, for example, program instructions or sets of instructions and an operating system. For example, program instructions and an operating system may be executed by the processing subsystem during operation of the cross-render multiview camera 100 . Note that one or more computer programs may constitute a computer program mechanism, a computer-readable storage medium, or software. Further, the instructions in the various modules within the memory subsystem may be implemented in one or more of a high-level procedural language, an object-oriented programming language, and assembly or machine language. Further, according to various embodiments, a programming language may be compiled or interpreted, eg, configurable or configurable (which may be used interchangeably in this discussion), to be executed by a processing subsystem.

In various embodiments, the power subsystem may include one or more energy storage components (eg, batteries) configured to power other components within the cross-render multi-view camera 100 . A networking subsystem may include one or more devices and subsystems or modules that are coupled to and configured to communicate (ie, perform network operations) coupled to one or both of a wired and wireless network. For example, the networking subsystem may include a Bluetooth ^TM networking system, a cellular networking system (eg, 3G/4G/5G networks such as UMTS, LTE, etc.), a universal serial bus (USB), IEEE networking systems based on standards described in 802.12 (eg, Wi-Fi networking systems), Ethernet networking systems, any or all.

Note that although some of the operations in the above-described embodiments may be implemented in hardware or software, the operations in the above-described embodiments may generally be implemented in a wide variety of configurations and architectures. Accordingly, some or all of the operations in the above-described embodiments may be performed by hardware, software, or both. For example, at least some of the operations in display technology may be implemented using program instructions, an operating system (eg, a driver for a display subsystem), or hardware.

According to other embodiments of the principles described herein, a cross-render multiview system is provided. 4 shows a block diagram of a cross-render multiview system 200 as an example in accordance with an embodiment consistent with the principles described herein. A cross-render multiview system 200 may be used to capture or image a scene 202 . For example, the image may be a multiview image 208 . Further, according to various embodiments, the cross-render multiview system 200 may be configured to display a multiview image 208 of the scene 202 .

As shown in FIG. 4 , a cross-render multiview system 200 includes a multiview camera array 210 having cameras spaced apart from each other along a first axis. According to various embodiments, the multiview camera array 210 is configured to capture a plurality of images 204 of the scene 202 . In some embodiments, the multi-view camera array 210 may be substantially similar to the plurality of cameras 110 described above with respect to the cross-render multi-view camera 100 . In particular, the multi-view camera array 210 may include a plurality of cameras arranged in a linear configuration along a first axis. In some embodiments, the multiview camera array 210 may include cameras that are not on the first axis.

The cross-render multiview system 200 shown in FIG. 4 further includes an image synthesizer 220 . The image synthesizer 220 may be configured to generate a composite image 206 of the scene 202 . In particular, the image synthesizer is configured to generate the composite image 206 using the difference map determined from the images 204 of the plurality of images. In some embodiments, the image synthesizer 220 may be substantially similar to the image synthesizer 120 of the cross-render multi-view camera 100 described above. For example, image synthesizer 220 may be configured to determine a difference map from which composite image 206 is generated. Also, image synthesizer 220 may provide hole filling to one or both of difference map and composite image 206 .

As shown, the cross-render multiview system 200 further includes a multiview display 230 . The multiview display 230 is configured to display the multiview image 208 of the scene 202 including the composite image 206 . According to various embodiments, the composite image 206 represents a view of the scene 202 from a viewpoint corresponding to the position of the virtual camera on a second axis orthogonal to the first axis. Additionally, the multiview display 230 may include the composite image 206 as a view of the multiview image 208 of the scene 202 . In some embodiments, the multiview image 208 comprises a plurality of composite images 206 corresponding to a plurality of virtual cameras and representing a plurality of different views of the scene 202 from a plurality of different viewpoints that are similar. may include In other embodiments, the multiview image 208 may include one or more images 204 of a plurality of images and a composite image 206 . For example, as shown in FIG. 3B , the multiview image 208 may include four views 4V, of which the first two views are a pair of composite images 206 . , and the second two views of the four views may be a pair of images 204 of the plurality of images.

In some embodiments, the plurality of cameras may include a pair of cameras in the multiview camera array 210 configured to provide a stereo pair of images 204 of the scene 202 . In such embodiments, the difference map may be determined by image synthesizer 220 using the stereo pair of images. In some embodiments, image synthesizer 220 is configured to provide a pair of composite images 206 of scene 202 . In such embodiments, the multiview image 208 may include a pair of composite images 206 . In some embodiments, the multiview image 208 may further include a pair of images 204 of a plurality of images.

In some embodiments, image synthesizer 220 may be implemented as a remote processor. For example, the remote processor may be a processor of a cloud computing service or a so-called 'cloud' processor. When the image synthesizer 220 is implemented as a remote processor, a plurality of images 204 may be sent to the remote processor by the cross-render multiview system, and then the composite image 206 is converted into the cross-render multiview system. may be received from the remote processor and displayed using the multi-view display 230 . According to various embodiments, transmission to or from a remote processor may use the Internet or a similar transmission medium. In other embodiments, the image synthesizer 220 may be implemented using another processor, such as, but not limited to, a processor (eg, GPU) of the cross-render multiview system 200 . According to other embodiments, a dedicated hardware circuit (eg, ASIC) of the cross-render multi-view system 200 may be used to implement the image synthesizer 220 .

According to various embodiments, the multi-view display 230 of the cross-render multi-view system 200 may be substantially any display or multi-view display capable of displaying a multi-view image. In some embodiments, multi-view display 230 may be a multi-view display that uses directional scattering of light and subsequent modulation of the scattered light to provide or display a multi-view image.

5A shows a cross-sectional view of a multiview display 300 as an example in accordance with an embodiment consistent with the principles described herein. 5B shows a top view of a multiview display 300 as an example in accordance with an embodiment consistent with the principles described herein. 5C shows a perspective view of a multiview display 300 as an example in accordance with an embodiment consistent with the principles described herein. The perspective view of FIG. 5C is only partially cut away to facilitate discussion herein. According to some embodiments, the multi-view display 300 may be used as the multi-view display 230 of the cross-render multi-view system 200 .

The multiview display 300 shown in FIGS. 5A-5C is configured to provide (eg, as a light field) a plurality of directional lightbeams 302 having different principal angular directions from each other. In particular, according to various embodiments, the provided plurality of directional lightbeams 302 corresponds to or equivalently to respective viewing directions of the multiview display 300 in the multiview image displayed by the multiview display 300 . is configured to scatter away from the multiview display 300 in different principal angular directions corresponding to directions of different views of the multiview image 208 of the cross-render multiview system 200 (eg, the cross-render multiview system 200 ). . According to various embodiments, the directional lightbeams 302 may be modulated (eg, using light valves as described below) to facilitate display of multiview content, ie information having a multiview image 208 . can 5A-5C also show a multi-view pixel 306 comprising an array of sub-pixels and light valves 330, which will be described below.

5A-5C , the multi-view display 300 includes a light guide 310 . The light guide 310 is configured to guide light along the length of the light guide 310 as guided light 304 (ie, a guided light beam). For example, the light guide 310 may include a dielectric material configured as an optical waveguide. The dielectric material may have a first index of refraction greater than a second index of refraction of a medium surrounding the dielectric optical waveguide. For example, the difference in refractive indices is configured to facilitate total internal reflection of the guided light 304 according to one or more guiding modes of the light guide 310 .

In some embodiments, light guide 310 may be an elongated, optically transparent substantially planar sheet, a slab or plate optical waveguide (ie, plate light guide) comprising a dielectric material. The dielectric material of the substantially planar sheet is configured to guide the guided light 304 using total internal reflection. According to various examples, the optically transparent material of the light guide 310 may be various types of glass (eg, silicon glass, alkali-aluminosilicate glass, borosilicate glass). glass, etc.) and substantially optically transparent plastics or polymers (eg, poly(methyl methacrylate) or 'acrylic glass', polycarbonate, etc.) ) may consist of or comprise any of a variety of dielectric materials including, but not limited to, one or more of In some examples, the light guide 310 further includes a cladding layer (not shown) on at least a portion of a surface (eg, one or both of a top surface and a bottom surface) of the light guide 310 . can do. According to some examples, a cladding layer may be used to further facilitate total internal reflection.

Further, in accordance with some embodiments, the light guide 310 includes a first surface 310 ′ (eg, a 'front' or anterior) and a second surface 310 ″ (eg, a front surface) of the light guide 310 . , the 'back' face or back) is configured to guide the guided light 304 according to total internal reflection at a non-zero propagation angle between It is guided and propagated by being reflected or 'bouncing' between the first surface 310 ′ and the second surface 310 ″ of the 310 . In some embodiments, a plurality of guided lightbeams of guided light 304 comprising different colors of light may be guided by light guide 310 at different per-color, non-zero propagation angles, respectively. Note that non-zero propagation angles are not shown in FIGS. 5A to 5C for simplicity of illustration. However, the bold arrow depicting the direction of propagation 303 in FIG. 5A shows the general direction of propagation of the guided light 304 along the length of the light guide.

As defined herein, a 'non-zero propagation angle' is a surface of the light guide 310 (eg, first surface 310 ′ or second surface 310 ″). Also, according to various embodiments, the non-zero propagation angle is greater than zero and less than the critical angle of total internal reflection within the light guide 310. For example, the non-zero propagation angle of the guided light 304 is The propagation angle may be between about 10 degrees and about 50 degrees, or in some examples between about 20 degrees and about 40 degrees, or between about 25 degrees and about 35 degrees For example, a non-zero propagation angle is about 30 In other examples, the non-zero propagation angle may be about 20 degrees, or about 25 degrees, or about 35 degrees, which is also a certain zero, as long as it is selected to be less than the critical angle of total internal reflection within light guide 310 . A propagation angle other than n may be chosen (eg, arbitrarily) for a particular implementation.

Guided light 304 within light guide 310 may enter or couple into light guide 310 at a non-zero propagation angle (eg, between about 30 degrees and about 35 degrees). In some examples, coupling structures such as, but not limited to, gratings, lenses, mirrors or similar reflectors (eg, inclined collimated reflectors), diffraction gratings and prisms (not shown), as well as various combinations thereof This guided light 304 may facilitate coupling light into the interior of the input end of the light guide 310 at a non-zero propagation angle. In other examples, light may enter directly into the input end of the light guide 310 without or substantially without the use of a coupling structure (ie, a direct or 'butt' coupling is used). can be). When coupled into the light guide 310 , the guided light 304 (eg, as a guided light beam) generally travels in a propagation direction 303 (eg, x in FIG. 5A ) that may be away from the input end. -shown as bold arrows pointing along the axis) are configured to propagate along the light guide 310 .

Also, according to various embodiments, the guided light 304 generated by coupling the light into the light guide 310 , or equivalently the guided light beam, may be a collimated light beam. As used herein, 'collimated light' or 'collimated light beam' is generally a beam of light in which the rays of the light beam are substantially parallel to each other within a light beam (eg, a guided light beam). is defined as Also by definition herein, rays of light that diverge or are scattered from a collimated lightbeam are not considered to be part of the collimated lightbeam. In some embodiments (not shown), the multiview display 300 may include a collimator (eg, an oblique collimator) such as a grating, lens, reflector, or mirror as described above, for example, to collimate light from a light source. reflectors). In some embodiments, the light source itself comprises a collimator. In either case, the collimated light provided to the light guide 310 is a collimated guided light beam. In various embodiments, the guided light 304 may be collimated according to or have a collimation coefficient σ. Alternatively, in other embodiments, the guided light 304 may be non-collimated.

In some embodiments, the light guide 310 may be configured to 'recycle' the guided light 304 . In particular, guided light 304 that has been guided along the length of the light guide may be redirected back along the length of the light guide in a different propagation direction 303 ′ than the propagation direction 303 . For example, the light guide 310 may include a reflector (not shown) at an end of the light guide 310 opposite an input end adjacent to the light source. The reflector may be configured to reflect the guided light 304 back towards the input end as recycled guided light. In some embodiments, instead of or in addition to light recycling (eg, using a reflector), another light source may provide light 304 directed in another propagation direction 303 ′. One or both of the recycling of the guided light 304 and the use of another light source to provide the guided light 304 having a different propagation direction 303 ′ can cause the guided light to be converted into a multi-channel light source, for example as described below. To the beam elements, making them available more than once may increase the brightness of the multiview display 300 (eg, increase the intensity of the directional lightbeams 302 ).

In FIG. 5A , a bold arrow (eg, directed in the negative x-direction) indicating the direction of propagation 303 ′ of the recycled guided light shows the general direction of propagation of the recycled guided light within the light guide 310 . do. Alternatively (eg, as opposed to recycling the guided light), guided light 304 propagating in another propagation direction 303 ′ may direct light in another propagation direction 303 ′ to the light guide 310 . ) by introducing it inside (eg, in addition to guided light 304 having a propagation direction 303 ).

5A-5C , the multi-view display 300 further includes an array of multi-beam elements 320 spaced apart from each other along the length of the light guide. In particular, the multi-beam elements 320 of the multi-beam element array are separated from each other by a finite space, and represent individual and distinct elements along the length of the light guide. That is, according to the definition of the present specification, the multi-beam elements 320 of the multi-beam element array are spaced apart from each other according to a finite (ie, non-zero) inter-element distance (eg, a finite center-to-center distance). have. Also, according to some embodiments, the multibeam elements 320 of the plurality of multibeam elements generally do not intersect, overlap, or otherwise contact each other. That is, each multi-beam element 320 of the plurality of multi-beam elements is generally distinct and spaced apart from the others of the multi-beam elements 320 .

According to some embodiments, the multi-beam elements 320 of the multi-beam element array may be arranged in a 1D array or a 2D array. For example, the multi-beam elements 320 may be arranged in a linear 1D array. In another example, the multibeam elements 320 may be arranged in a rectangular 2D array or a circular 2D array. Also, in some examples, the array (ie, 1D array or 2D array) may be a regular or uniform array. In particular, the inter-element distance (eg, center-to-center distance or spacing) between the multibeam elements 320 may be substantially uniform or constant across the array. In other examples, the inter-element distance between the multibeam elements 320 may vary across the array, along the length of the light guide 310 , or for both.

According to various embodiments, the multibeam element 320 of the multibeam element array may be configured to provide, couple out, or scatter a portion of the guided light 304 as a plurality of directional lightbeams 302 . For example, according to various embodiments, a portion of the guided light may be coupled out or scattered using one or more of diffractive scattering, specular scattering, and refractive scattering or coupling. 5A and 5C illustrate the directional lightbeams 302 as a plurality of divergent arrows depicted facing away from the first (or front) surface 310 ′ of the light guide 310 . Also, as defined above and as will be described later and as shown in FIGS. 5A-5C , according to various embodiments, the size of the multi-beam device 320 is a sub-pixel of the multi-view pixel 306 . (or equivalently the size of light valve 330 ). As used herein, 'size' may be defined in any of a variety of ways including, but not limited to, length, width, or area. For example, the size of the sub-pixel or light valve 330 may be their length, and a similar size of the multi-beam element 320 may also be the length of the multi-beam element 320 . In another example, size may refer to an area, and the area of multibeam element 320 may be similar to the area of a sub-pixel (or equivalently light valve 330 ).

In some embodiments, the size of the multibeam element 320 is similar to the sub-pixel size, and the multibeam element size may be between about 50% and about 200% of the sub-pixel size. For example, if the size of the multi-beam element is denoted by 's' and the size of the sub-pixel is denoted by 'S' (eg, as shown in FIG. 5A ), the size (s) of the multi-beam element is can be given as

In another example, the size of the multibeam element is greater than about 60% of the size of the sub-pixel, or greater than about 70% of the size of the sub-pixel, or greater than about 80% of the size of the sub-pixel, or the size of a sub-pixel. may be within a range greater than about 90% of the sub-pixel size, and less than about 180% of the sub-pixel size, or less than about 160% of the sub-pixel size, or less than about 140% of the sub-pixel size, or may be within a range of less than about 120% of the sub-pixel size. For example, according to 'comparable size', the size of the multi-beam element may be between about 75% and about 150% of the sub-pixel size. In another example, the multibeam element 320 may be similar in size to a sub-pixel, wherein the size of the multibeam element is between about 125% and about 85% of the size of the sub-pixel. According to some embodiments, similar sizes of multi-beam element 320 and sub-pixel may be selected to reduce, or in some instances minimize, dark zones between views of a multi-view display. . Also, similar sizes of multi-beam element 320 and sub-pixel may be selected to reduce, in some instances, minimize overlap between views (or view pixels) of a multi-view display.

The multiview display 300 shown in FIGS. 5A-5C further includes an array of light valves 330 configured to modulate the directional lightbeams 302 of the plurality of directional lightbeams. In various embodiments, various types of light valves, including, but not limited to, one or more of liquid crystal light valves, electrophoretic light valves, and electrowetting based light valves may be used as light valves in a light valve array. 330 may be used.

5A-5C , different ones of the directional lightbeams 302 with different principal angular directions pass through and are modulated by different ones of the light valves 330 in the light valve array. Also, as shown, a light valve 330 of the array corresponds to a sub-pixel of a multiview pixel 306 , and a set of light valves 330 is connected to a multiview pixel 306 of a multiview display. respond In particular, as shown, a different set of light valves 330 of the light valve array are configured to receive and modulate directional lightbeams 302 from a corresponding one of the multibeam elements 320 , ie, each There is one unique set of light valves 330 per multibeam element 320 .

As shown in FIG. 5A , the first set of light valves 330a is configured to receive and modulate the directional lightbeams 302 from the first multibeam element 320a. The second set of light valves 330b is also configured to receive and modulate the directional lightbeams 302 from the second multibeam element 320b. Accordingly, as shown in FIG. 5A , each of the light valve sets in the light valve array (eg, the first and second light valve sets 330a and 330b ) are each different multibeam element 320 ( For example, corresponding to both elements 320a , 320b ) and a different multiview pixel 306 , the individual light valves 330 of the light valve sets are a sub-pixel of each multiview pixel 306 . respond to

Note that, as shown in FIG. 5A , the size of the sub-pixels of the multiview pixel 306 may correspond to the size of the light valve 330 in the light valve array. In other examples, the size of a sub-pixel may be defined as the distance (eg, center-to-center distance) between adjacent light valves 330 of the light valve array. For example, the light valves 330 may be less than the center-to-center distance between the light valves 330 in the light valve array. For example, the size of the sub-pixel may be defined as a size corresponding to the size of the light valve 330 or a center-to-center distance between the light valves 330 .

In some embodiments, the relationship between multibeam elements 320 and corresponding multiview pixels 306 (ie, sets of sub-pixels and corresponding sets of light valves 330 ) is a one-to-one relationship. can That is, the number of multi-view pixels 306 and the number of multi-beam elements 320 may be the same. 5B explicitly illustrates by way of example a one-to-one relationship in which each multiview pixel 306 comprising a different set of light valves 330 (and corresponding sub-pixels) is shown surrounded by a dashed line. In other embodiments (not shown), the number of multi-view pixels 306 and the number of multi-beam elements 320 may be different from each other.

In some embodiments, the inter-element distance (eg, center-to-center distance) between a pair of multibeam elements 320 of the plurality of multibeam elements corresponds to, for example, indicated by light valve sets. may be equal to an inter-pixel distance (eg, a center-to-center distance) between a pair of multi-view pixels 306 . For example, as shown in FIG. 5A , the center-to-center distance d between the first multi-beam element 320a and the second multi-beam element 320b is the first light valve set 330a and the second light valve set 330a. It may be substantially equal to the center-to-center distance D between sets 330b. In other embodiments (not shown), the relative center-to-center distances of pairs of multi-beam elements 320 and corresponding light valve sets may be different. For example, the multibeam elements 320 may have an inter-element spacing (ie, center-to-center distance d) greater or less than the spacing between sets of light valves representing multiview pixels 306 (ie, center-to-center distance D). can have

In some embodiments, the shape of the multi-beam element 320 is the shape of the multi-view pixel 306 , or, equivalently, a set of light valves 330 (or 'sub-') corresponding to the multi-view pixel 306 . The shape of the array') is similar. For example, the multibeam element 320 may have a square shape, and the multiview pixel 306 (or a corresponding arrangement of a set of light valves 330 ) may be substantially square. In another example, the multibeam element 320 may have a rectangular shape, ie, a length or longitudinal dimension that is greater than a width or transverse dimension. In this example, the multiview pixel 306 (or equivalent arrangement of a set of light valves 330 ) corresponding to the multibeam element 320 may have a similar rectangular shape. FIG. 5B shows a top view of a square-shaped multi-beam element 320 and corresponding square-shaped multi-view pixels 306 comprising square sets of light valves 330 . In still other examples (not shown), the multibeam elements 320 and corresponding multiview pixels 306 may have a variety of shapes, including or at least approximate to triangular, hexagonal and circular shapes. However, it is not limited thereto.

Also (eg, as shown in FIG. 5A ), in accordance with some embodiments, each multi-beam element 320 is assigned at a given time based on a set of sub-pixels assigned to a particular multi-view pixel 306 . is configured to provide directional lightbeams 302 to only one multiview pixel 306 . In particular, with respect to the assignment of a set of sub-pixels to a given one of the multi-beam elements 320 and to a particular multi-view pixel 306 , as shown in FIG. 5A , corresponding to different views of the multi-view display. The directional lightbeams 302 with different main angular directions are directed to one corresponding multiview pixel 306 and its sub-pixels, ie a set of light valves 330 corresponding to the multibeam element 320 . , is practically limited. As such, each multibeam element 320 of the multiview display 300 has a corresponding set of directional lightbeams 302 having a set of different principal angular directions corresponding to different views of the multiview display 300 . ) (ie, a set of directional lightbeams 302 includes a lightbeam having a direction corresponding to each of the different viewing directions).

As illustrated, the multi-view display 300 may further include a light source 340 . According to various embodiments, the light source 340 may be configured to provide light to be guided within the light guide 310 . In particular, the light source 340 may be positioned adjacent to an entrance surface or end (input end) of the light guide 310 . In various embodiments, the light source 340 includes substantially any source of light (eg, an optical emitter) including, but not limited to, an LED, a laser (eg, a laser diode), or a combination thereof. may include In some embodiments, light source 340 may include an optical emitter configured to produce substantially monochromatic light having a narrowband spectrum appearing in a particular color. In particular, the color of monochromatic light may be a primary color of a particular color space or color model (eg, a red-green-blue (RGB) color model). In other examples, light source 340 may be a substantially broadband light source configured to provide substantially broadband or polychromatic light. For example, the light source 340 may provide white light. In some embodiments, light source 340 may include a plurality of different optical emitters configured to provide different colors of light. The different optical emitters may be configured to provide light having different, per-color, non-zero propagation angles of the guided light corresponding to each of the different colors of light.

In some embodiments, the light source 340 may further include a collimator. The collimator may be configured to receive substantially uncollimated light from one or more of the optical emitters of the light source 340 . Further, the collimator may be configured to convert substantially non-collimated light to collimated light. In particular, according to some embodiments, the collimator may provide collimated light that has a non-zero propagation angle and is collimated according to a predetermined collimation coefficient σ. Further, when optical emitters of different colors are used, the collimator may be configured to provide collimated light having one or both of different, per-color, non-zero propagation angles and different per-color collimation coefficients. The collimator is also configured to deliver a collimated light beam to the light guide 310 so that it can propagate as the guided light 304 described above.

In some embodiments, the multi-view display 300 may display light in a direction passing through the light guide 310 that is orthogonal (or substantially orthogonal) to the propagation directions 303 , 303 ′ of the guided light 304 . substantially transparent. In particular, in some embodiments, the light guide 310 and spaced apart multibeam elements 320 allow light through both the first surface 310 ′ and the second surface 310 ″ to pass through the light guide 310 . allow it to pass Transparency is at least in part due to the relatively small size of the multi-beam elements 320 and the relatively large inter-element spacing of the multi-beam elements 320 (eg, a one-to-one correspondence with the multi-view pixels 306 ). ) can be facilitated due to both. Further, according to some embodiments, the multi-beam elements 320 may be substantially transparent to light propagating orthogonally to the surfaces 310 ′ and 310 ″ of the light guide body.

According to various embodiments, comprising diffraction gratings, micro-reflective elements and/or micro-refractive elements optically coupled to a light guide 310 to scatter the guided light beam 304 as directional light beams 302 . A wide variety of optical components may be used to generate the directional lightbeams 302 . These optical components may be positioned on the first surface 310 ′, the second surface 310 ″ or even between the first and second surfaces 310 ′, 310 ″ of the light guide 310 . take note of Further, according to some embodiments, the optical component is a 'positive feature' that protrudes from the first surface 310' or the second surface 310'', or the first surface 310' or It may be a 'negative feature' concave from the second surface 310 ″.

In some embodiments, light guide 310 , multibeam elements 320 , light source 340 and/or an optional collimator may function as a multiview backlight. Such a multi-view backlight may be used with a light valve array of the multi-view display 300 , for example as the multi-view display 230 . For example, the multi-view backlight may be configured by the light valves 330 that modulate the directional light beams 302 provided by the multi-view backlight to provide directional views of the multi-view image 208 , as described above. It can serve as a source of light to the array (often as a panel backlight).

In some embodiments, the multi-view display 300 may further include a wide-angle backlight. In particular, the multi-view display 300 (or the multi-view display 230 of the cross-render multi-view system 200 ) may include a wide-angle backlight in addition to the multi-view backlight described above. For example, a wide-angle backlight may be adjacent to a multi-view backlight.

6 shows a cross-sectional view of a multiview display 300 including a wide-angle backlight 350 as an example in accordance with one embodiment consistent with the principles described herein. As shown, the wide-angle backlight 350 is configured to provide a broad-angle emitted light 352 during a first mode. According to various embodiments, the multi-view backlight (eg, light guide 310 , multibeam elements 320 , and light source 340 ) transmits the directional emitted light as directional lightbeams 302 during the second mode. can be configured to provide. Further, the array of light valves is configured to modulate the wide angle emission light 352 to provide a two-dimensional (2D) image during a first mode, and a directional emission light (or directional light) to provide a multi-view image during the second mode. and modulate the light beams 302). For example, when the multi-view display 300 shown in FIG. 6 is used as the multi-view display 230 of the cross-render multi-view system 200 , the cameras of the multi-view camera array 210 or the cameras A two-dimensional (2D) image may be captured. As such, according to some embodiments, the 2D image may simply represent one of the directional views of the scene 202 during the second mode.

As shown on the left side of FIG. 6 , a multiview image MULTIVIEW uses a light source 340 to provide directional lightbeams 302 that are scattered from a light guide 310 using multibeam elements 320 . By activating it, it can be provided using a multi-view backlight. Alternatively, as shown on the right side of FIG. 5 , the 2D image deactivates the light source 340 and activates the wide angle backlight 350 to provide the wide angle emission light 352 to the array of light valves 330 . It can be provided by As such, according to various embodiments, the multi-view display 300 including the wide-angle backlight 350 may be switched between the display of the multi-view image and the display of the 2D image.

According to other embodiments of the principles described herein, a cross-render multiview imaging method is provided. 7 shows a flow diagram of a cross-render multiview imaging method 400 as an example in accordance with an embodiment consistent with the principles described herein. As shown in FIG. 7 , a cross-render multiview imaging method 400 includes capturing 410 a plurality of images of a scene using a plurality of cameras spaced apart from each other along a first axis. In some embodiments, the plurality of images and the plurality of cameras may be substantially similar to the plurality of images 104 and the plurality of cameras 110 of the cross-render multiview camera 100 , respectively. Likewise, in some embodiments, a scene may be substantially similar to scene 102 .

The cross-render multi-view imaging method 400 shown in FIG. 7 further includes generating 420 a composite image of the scene using the scene difference map determined from the plurality of images. According to various embodiments, the composite image represents a view of the scene from a viewpoint corresponding to the position of the virtual camera on the second axis displaced from the first axis. In some embodiments, the image synthesizer may be substantially similar to the image synthesizer 120 of the cross-render multi-view camera 100 described above. In particular, according to various embodiments, the image synthesizer may determine a difference map from images of a plurality of images.

In some embodiments (not shown), the cross-render multiview imaging method 400 may further include hole filling in one or both of the difference map and the composite image. For example, hole filling may be implemented by an image synthesizer.

In some embodiments, the plurality of cameras may include a pair of cameras configured to capture a stereo pair of images of a scene. In such embodiments, the difference map may be determined using the stereo pair of images. Also, generating 420 of the composite image may generate a plurality of composite images representing views of the scene from viewpoints corresponding to positions of a plurality of similar virtual cameras.

In some embodiments (not shown), the cross-render multi-view imaging method 400 further comprises displaying the composite image as a view of the multi-view image using the multi-view display. In particular, the multi-view image may include one or more composite images representing different views of the multi-view image displayed by the multi-view display. In addition, the multi-view image may include views representing one or more of the plurality of images. For example, the multi-view image may include a stereo pair of composite images as shown in FIG. 3A , or a stereo pair of composite images as shown in FIG. 3B and a pair of images from a plurality of images. may include In some embodiments, the multiview display may be substantially similar to the multiview display 230 of the cross-render multiview system 200 described above, or may be substantially similar to the multiview display 300 described above. have.

In the above, examples and embodiments of a cross-render multi-view camera, a cross-render multi-view system, and a cross-render multi-view imaging method that provide a composite image from a difference/depth map of images captured by a plurality of cameras are provided. explained. It is to be understood that the foregoing examples are merely illustrative of some of the many specific examples that are representative of the principles described herein. Obviously, a person skilled in the art can readily devise numerous other configurations without departing from the scope defined by the following claims.

Claims (20)

A cross-render multiview camera comprising:
a plurality of cameras spaced apart from each other along a first axis and configured to capture a plurality of images of the scene; and
an image synthesizer configured to generate a composite image of the scene using a disparity map of the scene determined from the plurality of images;
the composite image represents a view of the scene from a viewpoint corresponding to the position of the virtual camera on a second axis displaced from the first axis
Cross-render multi-view camera.
The method of claim 1,
The second axis is perpendicular to the first axis
Cross-render multi-view camera.
The method of claim 1,
the image synthesizer is configured to provide a plurality of composite images using the difference map;
each composite image of the plurality of composite images represents a view of the scene from a different viewpoint of the scene compared to other composite images of the plurality of composite images
Cross-render multi-view camera.
The method of claim 1,
The plurality of cameras includes a pair of cameras configured as stereo cameras,
the plurality of images of the scene captured by the stereo camera comprises a stereo pair of images of the scene;
the image synthesizer is configured to provide a plurality of composite images representing views of the scene from viewpoints corresponding to positions of a plurality of virtual cameras
Cross-render multi-view camera.
5. The method of claim 4,
the first axis is a horizontal axis,
The second axis is a vertical axis orthogonal to the horizontal axis,
the stereo pairs of images are arranged in a horizontal direction corresponding to the horizontal axis,
The plurality of composite images includes a pair of composite images arranged in a vertical direction corresponding to the vertical axis.
Cross-render multi-view camera.
The method of claim 1,
The image synthesizer is further configured to provide hole-filling to the difference map.
Cross-render multi-view camera.
A cross-render multiview system comprising the cross-render multiview camera of claim 1, comprising:
a multi-view display configured to display the composite image as a view of the multi-view image representing the scene
Cross-render multiview system.
8. The method of claim 7,
The multi-view display is further configured to display the plurality of images from cameras of the plurality of cameras as different views of the multi-view image.
Cross-render multiview system.
A cross-render multiview system comprising:
a multiview camera array having cameras spaced apart from each other along a first axis and configured to capture a plurality of images of the scene;
an image synthesizer configured to generate a composite image of the scene using a difference map determined from the plurality of images; and
a multi-view display configured to display a multi-view image of the scene including the composite image;
The composite image represents a view of the scene from a viewpoint corresponding to a virtual camera located on a second axis orthogonal to the first axis.
Cross-render multiview system.
10. The method of claim 9,
wherein the multiview camera array comprises a pair of cameras configured to provide a stereo pair of images of the scene;
The difference map is determined by the image synthesizer using the stereo pair of images.
Cross-render multiview system.
10. The method of claim 9,
the image synthesizer is configured to provide a pair of composite images of the scene;
The multi-view image includes a pair of images among the plurality of images and the pair of composite images
Cross-render multiview system.
10. The method of claim 9,
The image synthesizer is implemented as a remote processor,
the plurality of images are transmitted to the remote processor by the cross-render multiview system;
wherein the composite image is received from the remote processor by the cross-render multiview system and displayed using the multiview display.
Cross-render multiview system.
10. The method of claim 9,
The multi-view display,
a light guide configured to guide light;
an array of multibeam elements spaced apart from each other and configured to scatter guided light from the light guide as directional lightbeams having directions corresponding to viewing directions of the multiview image; and
a light valve array configured to modulate the directional lightbeams to provide the multiview image;
The multi-beam element of the array of multi-beam elements has a size similar to that of a light valve of the light valve array and a shape similar to that of a multi-view pixel associated with the multi-beam element.
Cross-render multiview system.
14. The method of claim 13,
wherein the multibeam element of the array of multibeam elements comprises at least one of a diffraction grating, a microreflective element and a microrefractive element optically coupled to the light guide for scattering the guided light as the directional lightbeams.
Cross-render multiview system.
14. The method of claim 13,
wherein the multi-view display further comprises a light source optically coupled to the input of the light guide;
wherein the light source is configured to provide the guided light corresponding to one or both of having a non-zero propagation angle and collimating according to a predetermined collimation coefficient.
Cross-render multiview system.
14. The method of claim 13,
wherein the multi-view display further comprises a wide-angle backlight configured to provide a wide-angle emission light during the first mode;
the light guide and the array of multibeam elements are configured to provide the directional lightbeams during a second mode;
wherein the light valve array is configured to modulate the wide angle emission light to provide a two-dimensional image during the first mode and modulate the directional light beams to provide the multi-view image during the second mode
Cross-render multiview system.
A cross-render multi-view imaging method comprising:
capturing a plurality of images of the scene using a plurality of cameras spaced apart from each other along a first axis; and
generating a composite image of the scene using a difference map of the scene determined from the plurality of images;
the composite image represents a view of the scene from a viewpoint corresponding to the position of the virtual camera on a second axis displaced from the first axis
Way.
18. The method of claim 17,
providing hole-filling to the difference map.
Way.
18. The method of claim 17,
wherein the plurality of cameras comprises a pair of cameras configured to capture a stereo pair of images of the scene;
the difference map is determined using the stereo pair of images,
Generating the composite image may include generating a plurality of composite images representing views of the scene from viewpoints corresponding to positions of a plurality of similar virtual cameras.
Way.
18. The method of claim 17,
using a multi-view display to display the composite image as a view of the multi-view image.
Way.

Images