JP4483261B2 - Stereoscopic image processing device - Google Patents

Description

  The present invention relates to a stereoscopic image processing apparatus capable of generating a stereoscopic image.

  Conventionally, a stereoscopic image can be obtained by taking two still images using the parallax between the left and right eyes and displaying them so that they can be observed by the left and right eyes. The existence of a processing device is known.

  The two images are composed of a left viewpoint image (L image) based on the viewpoint from the left eye and a right viewpoint image (R image) based on the viewpoint from the right eye. Further, the L image and the R image are greatly different in size, image quality, and the like according to the performance of the optical adapter attached to the imaging apparatus.

  Therefore, the imaging device detects information related to the optical adapter such as a stereo adapter, and captures images so as to obtain appropriate viewpoint images according to the type of the optical adapter (see, for example, Patent Document 1). The technical literature information related to the present invention includes the following.

JP 2002-214515 A

  However, since the captured image such as the viewpoint image does not include information about the optical adapter at the time of shooting, a stereoscopic effect can be obtained even if the stereoscopic image processing device converts each viewpoint image into a stereoscopic image. It was difficult to convert it into a 3D image.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a new and improved stereoscopic image processing apparatus capable of including adapter information relating to an optical adapter in an image. It is.

  In order to solve the above problems, according to a first aspect of the present invention, there is provided a stereoscopic image processing apparatus that manages at least a plurality of viewpoint images having parallax as one combined image. By the stereoscopic image processing apparatus, the tag information of the combined image includes adapter information related to the optical adapter of the imaging apparatus in which a plurality of viewpoint images are captured.

  According to the present invention, the stereoscopic image processing device generates a stereoscopic image based on a combined image composed of a plurality of viewpoint images. The stereoscopic image processing apparatus acquires adapter information included in the tag information of the combined image when generating the stereoscopic image. According to such a configuration, a stereoscopic image utilizing the characteristics of the optical adapter can be generated by referring to the adapter information related to the optical adapter included in the tag information. A stereoscopic image is a general term for images that can be displayed stereoscopically.

  The stereoscopic image processing apparatus may be configured to determine effective areas of a plurality of viewpoint images based on adapter information. With such a configuration, a stereoscopic image with high visibility can be obtained by setting an effective region suitable for a stereoscopic image, except for the unnecessary region of the combined image that is not suitable for a stereoscopic image because the stereoscopic effect cannot be obtained. Can be generated.

  The stereoscopic image processing apparatus may be configured to determine the brightness of a plurality of viewpoint images based on the adapter information.

  The stereoscopic image processing apparatus may be configured to determine resolutions of a plurality of viewpoint images based on adapter information. With this configuration, a stereoscopic image suitable for visual recognition can be generated.

  The stereoscopic image processing apparatus may be configured to correct chromatic aberration caused by color shift or color blur of the plurality of viewpoint images based on the adapter information.

  The optical adapter may be configured to be any one of at least a two-mirror adapter, a four-mirror adapter, a chevron prism adapter, and an anamorphic lens adapter. With this configuration, it is possible to generate a highly visible stereoscopic image for a wide variety of optical adapters.

  The correction of chromatic aberration can be configured such that an area where color misregistration or color blur occurs is filled with a correction color. With this configuration, it is possible to easily correct chromatic aberration and generate a highly visible stereoscopic image.

The correction color may be configured to be at least one or both of black and white. With this configuration, it is possible to generate a highly visible stereoscopic image while correcting chromatic aberration. The mixing ratio of black and white is arbitrary. The plurality of viewpoint images may be a right viewpoint image and a left viewpoint image. According to such a configuration, the stereoscopic image processing device generates a stereoscopic image based on the combined image composed of the right viewpoint image and the left viewpoint image. The stereoscopic image processing apparatus acquires adapter information included in the tag information of the combined image when generating the stereoscopic image. According to this configuration, a stereoscopic image utilizing the characteristics of the optical adapter can be generated by referring to the adapter information related to the optical adapter included in the tag information.

  The stereoscopic image processing apparatus may be configured to determine at least one effective area of the right viewpoint image or the left viewpoint image based on the adapter information. With this configuration, it is possible to generate a stereoscopic image with a high degree of visibility by setting an effective area suitable for a stereoscopic image except for an unnecessary area of the combined image that is not suitable for a stereoscopic image. Can do.

  The stereoscopic image processing apparatus may be configured to determine the luminance of at least one of the right viewpoint image and the left viewpoint image based on the adapter information.

  The stereoscopic image processing apparatus may be configured to determine the resolution of at least one of the right viewpoint image and the left viewpoint image based on the adapter information. With this configuration, it is possible to generate a stereoscopic image with an appropriate stereoscopic effect.

  The stereoscopic image processing apparatus may be configured to correct chromatic aberration caused by color shift or color blur of at least one of the right viewpoint image and the left viewpoint image based on the adapter information.

  The optical adapter may be configured to be any one of at least a two-mirror adapter, a four-mirror adapter, a chevron prism adapter, and an anamorphic lens adapter. With such a configuration, it is possible to generate a stereoscopic image having a stereoscopic effect and high visibility for a wide variety of optical adapters.

  The correction of chromatic aberration can be configured such that an area where color misregistration or color blur occurs is filled with a correction color. With such a configuration, it is possible to easily correct chromatic aberration and generate a stereoscopic image having a stereoscopic effect and high visibility.

The correction color may be configured to be at least one or both of black and white. With this configuration, it is possible to generate a highly visible stereoscopic image while correcting chromatic aberration. The mixing ratio of black and white is arbitrary. The right viewpoint image and the left viewpoint image may be managed as one combined image, and the combined image may be configured to be at least a JPEG format image. With such a configuration, adapter information related to the optical adapter can be efficiently managed.

  As described above, according to the present invention, since information regarding the optical adapter at the time of shooting can be included in image data such as a captured viewpoint image, appropriate correction is performed on each viewpoint image to create a stereoscopic image. It can be combined with a stereoscopic image that gives a feeling.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, components having substantially the same functions and configurations are denoted by the same reference numerals, and redundant description is omitted.

  There are a number of methods for stereoscopically viewing a two-dimensional image (hereinafter referred to as “stereoscopic viewing method”) using the spatial shift (binocular parallax) between retinal images acquired by the left and right eyes of a human or the like. . In addition, when there is no description about parallax, it shall show the parallax in a horizontal direction.

  As a method of stereoscopic viewing, an anaglyph method using special glasses, a color anaglyph method, a polarizing filter method, a time-division stereoscopic television method, and a lenticular method not using special glasses are known.

  In order to realize the various stereoscopic viewing methods described above, it is necessary to acquire a left viewpoint image (L image) for the left eye and a right viewpoint image (R image) for the right eye. In order to acquire the L image and the R image, the easiest method is to move the position of the camera by, for example, the distance between both eyes of the person and photograph the same subject twice.

  Further, as a method of acquiring the L image and the R image by one shooting, for example, as shown in FIGS. 1 and 2, an optical adapter 105 composed of a mirror or the like is placed outside the lens 103 of the imaging device 100. A method of attaching is known. The imaging apparatus 100 can be exemplified by a digital camera, for example.

  FIG. 2 schematically shows the structure of the optical adapter 105. The optical image for the right eye that enters from a single daylighting window is reflected by the mirror 121 toward the mirror 122, reflected by the mirror 122 toward the photographing lens 103, and then collected by the photographing lens 103. . The optical image for the left eye that enters from the single daylighting window is collected by the photographing lens 103 without being reflected by the mirror 121 and the mirror 122.

  The optical image incident through the optical adapter 105 is photographed as a parallax image including a viewpoint area from the left eye and a viewpoint area from the right eye. This left-eye area (left-eye image) is used as an L image, and the right-eye area (right-eye image) is used as an R image. In addition, since it is sufficient if the set of viewpoint images is a parallax image and there are a plurality of viewpoint images, it is not always necessary to configure them integrally on both sides.

  Next, as shown in FIG. 3, as a method of displaying a stereoscopic image (stereoscopic image, 3D image) obtained by synthesizing parallax images made up of the respective viewpoint images, for example, as shown in FIG. There is a polarization filter system.

  As shown in FIG. 4, the polarization filter method reflects the right eye projector 141 for projecting the R image, the left eye projector 142 for projecting the L image, and the light of the L image and the R image. The screen 143 and the polarizing glasses 144 are included. Note that a 3D (three-dimensional) image is a stereoscopic image that can be viewed stereoscopically.

  The right-eye projector 141 includes a polarizing filter in the vertical direction. The left-eye projector 142 includes a polarizing filter in the horizontal direction. Therefore, the light of the R image output from the right-eye projector 141 is the light of the horizontal arrow shown in FIG. The light of the L image output from the left-eye projector 142 is the light of the arrow in the vertical direction shown in FIG.

  Next, on the screen 143, the L image projected by the linearly polarized light in the vertical direction and the R image projected by the linearly polarized light in the horizontal direction are overlapped to generate a stereoscopic image.

  Then, by using polarizing glasses 144 in which a horizontal linear polarizing filter is arranged on the left side and a vertical linear polarizing filter is arranged on the right side, the right eye projector 141 out of the stereoscopic image reflected by the screen 143 is used. The projected R image only passes through the right linear polarizing filter, and the L image output from the left-eye projector 142 passes only through the left linear polarizing filter.

  Therefore, when the stereoscopic image on the screen 143 is viewed from the polarizing glasses 144, the image can be viewed stereoscopically, for example, a photographed building appears to pop out.

  In addition to the case where a stereoscopic image is generated on the screen 143 shown in FIG. 4, for example, a stereoscopic image may be generated and displayed by a personal computer (PC) or a computer device. Next, a case where a stereoscopic image is generated by the computer device according to the present embodiment will be described with reference to FIGS.

  As described above, a parallax image is generated by being reflected by the mirror 121 and the mirror 122 shown in FIG. The parallax image is composed of an L image and an R image as shown in FIG.

  Next, a computer apparatus applied to the stereoscopic image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 5 is an explanatory diagram showing a schematic configuration of the computer apparatus according to the present embodiment.

  The computer device 150 is an information processing device including at least a central processing unit (CPU) and a storage unit, and is generally a computer device, but is a portable terminal, a PDA (Personal Digital Assistant), a notebook personal computer, An information processing apparatus such as a desktop personal computer is also included.

  As shown in FIG. 5, the computer device 150 includes a computer device 150 that generates a stereoscopic image, polarized glasses 171 that a user uses when viewing the displayed stereoscopic image, and a stereoscopic that displays the stereoscopic image. A visual display unit 140 and a line polarizing plate 172 disposed outside the display surface of the stereoscopic display unit 140 are further provided.

  The polarized glasses 171 are supported by a support bar 170 attached to the computer apparatus 150 so as to be positioned in an upper space near the keyboard of the computer apparatus 150.

  Next, as shown in FIG. 6, the generated parallax images of the L image and the R image are combined with the L image and the R image according to the following equation (1), and a stereoscopic image as shown in FIG. Generated. The even lines and odd lines are horizontal columns configured in the stereoscopic display unit 140 provided in the computer device 150.

  For example, when the display unit is UXGA (Ultra etended Graphics Array), as shown in Expression (1), if the top end is the 0th line among the horizontal lines, the 0th line is an even line, Next, the first line is an odd-numbered line, and so on, and continues to the lowermost line (1599 line).

Even line 1 × (L image pixel) + 0 × (R image pixel) = (stereoscopic image pixel)
Odd line 0 × (L image pixel) + 1 × (R image pixel) = (stereoscopic image pixel)
... Formula (1)

  As shown in FIG. 7, the L image and the R image are synthesized for each horizontal line sequentially from the 0th line, thereby generating a stereoscopic image in which the L image and the R image are alternately synthesized for each line. Is done. The generated stereoscopic image is displayed on, for example, the stereoscopic display unit 140 provided in the computer device.

  As shown in FIG. 8, the user views the stereoscopic image displayed on the stereoscopic display unit 140 through the polarized glasses 171. The stereoscopic display unit 140 includes a line polarizing plate 172 in advance.

  The line polarizing plate 172 has a plurality of horizontal lines. Among the plurality of lines of the line polarizing plate 172, in order from the uppermost end, the even-numbered line includes a vertical polarizing plate, and the odd-numbered line includes a horizontal polarizing plate.

  The right side of the polarizing glasses 171 includes a horizontal polarizing filter, and the left side includes a vertical polarizing filter. Therefore, of the L image light or the R image light that has passed through the line polarizing plate 172, only the L image light consisting of odd lines passes through the left side of the polarizing glasses 171 and the R image consisting of odd lines passes through the right side. Only the light passes through. That is, the user can view the stereoscopic image stereoscopically.

(Stereoscopic image processing device)
Here, the stereoscopic image processing apparatus according to the present embodiment will be described with reference to FIG. FIG. 9 is a block diagram showing a schematic configuration of the stereoscopic image processing apparatus according to the present embodiment.

  As shown in FIG. 9, a stereoscopic image processing apparatus that generates a stereoscopic image that can be viewed stereoscopically includes an imaging unit 101, an image encoding unit 132, an image control information generation unit 133, and data multiplexing. The image forming unit 134, the recording medium 135, the data separating unit 136, the image decoding unit 137, the image separating unit 138, the image converting unit 139, and the stereoscopic display unit 140.

  The imaging unit 101 includes an imaging device (CCD) 130-1 and an imaging device 130-2 that image a subject, and a combining unit 131. It is also possible to configure the image pickup device 130-1 and the image pickup device 130-2 as a single unit by configuring the image pickup device 130-1 and the image pickup device 130-2. In the above case, it is possible to provide the optical adapter 105 or the like.

  The image of the viewpoint from the left eye (L image) captured by the image sensor 130-1 and the image of the viewpoint from the right eye (R image) captured by the image sensor 130-1 are combined by the combining unit 131. Is transmitted. Note that the stereoscopic image processing apparatus shown in FIG. 9 is described by taking the case of two viewpoints as an example, but is not limited to such an example, and can be implemented even in the case of a plurality of viewpoints.

  The synthesizing unit 131 rotates the transmitted viewpoint images (L image, R image) and synthesizes them to generate a combined image. Further, the synthesis unit 131 instructs the generated combined image to be recorded on the recording medium 135 or a memory (not shown). The combined image shown in FIG. 9 is such that the L image and the R image are adjacent to each other on the left and right, but is not limited to such an example. For example, the L image and the R image are adjacent to each other vertically. Also good. The combined image will be described later.

  The combined image is encoded by the image encoding unit 132. Examples of the encoding include a JPEG (Joint Photographic Experts Group) format.

  The image control information generation unit 133 generates tag information (hereinafter referred to as image control information) for converting the combined image into a stereoscopic image. The image control information includes imaging information such as exposure, date and time, presence / absence of flash, etc. at the time of imaging of the imaging unit 101, information for generating an appropriate stereoscopic image on the stereoscopic display unit 140, and the like. For example, information for displaying left and right images (L image, R image) for each line on the display screen is exemplified.

  The data multiplexing unit 134 multiplexes the combined image transmitted from the image encoding unit 132 and the image control information transmitted from the image control information generating unit 133. The multiplexed combined image and image control information are recorded on the recording medium 135.

  The recording medium 135 is a storage device capable of reading and writing data. For example, an HDD device (hard disk drive device), a CD-RW (Rewriteable), a DVD-RAM (DVD-Random Access Memory), an EEPROM (Electrically Erasable Programmable Read). For example, an only memory) or a memory stick (registered trademark) is exemplified.

  The data separation unit 136 acquires the combined image and the image control information recorded on the recording medium 135, respectively. The data separation unit 136 transmits the acquired combined image to the image decoding unit 137 and transmits image control information to the image separation unit 138. The combined image data and the image control information are recorded in a predetermined location (such as a folder) of the recording medium 135.

  Note that the data separation unit 136 according to the present embodiment has been described by taking as an example the case of acquiring the combined image and the image control information from the recording medium 135, but if the combined image and the image control information are linked, The present invention is not limited to this example. For example, the present invention can be implemented even when recording folders are different, or when combined images and image control information are recorded separately and acquired separately from a server on a network. is there.

  The image decoding unit 137 decodes the pre-encoded combined image data and transmits it to the image separation unit 138.

  The image separation unit 138 acquires the combined image specified in the image control information based on the image control information transmitted from the data separation unit 136, and separates it into images of each viewpoint (L image, R image). When separating the L image and the R image, an effective area of the L image and the R image of the combined image may be cut out and rotated or enlarged / reduced by a predetermined angle. As a result of the separation, the image separation unit 138 performs processing so that the orientations of the L image and the R image are the same. The effective area will be described later.

  The image conversion unit 139 converts the L image and the R image transmitted from the image separation unit 138 into a stereoscopic image by superimposing or the like as described above. Note that when a cutout area or the like is specified in the image control information, the image conversion unit 139 may convert the image into a stereoscopic image by performing a cutout process or an enlargement / reduction process. The cutout area will be described later.

  The stereoscopic display unit 140 displays the stereoscopic image synthesized by the image conversion unit 139. The stereoscopic display unit 140 is exemplified by a display unit such as a display provided in a computer device, which will be described later, or a projector device that projects and displays on a screen.

  Note that the stereoscopic display unit 140 according to the present embodiment is not limited to displaying a stereoscopic image. For example, when displaying a 2D still image, a moving image, or outputting a sound, Alternatively, the present invention can be implemented even when a printout unit is provided instead of the stereoscopic display unit 140 and an anaglyph stereoscopic image is output to a medium such as paper.

  Furthermore, a modification of the stereoscopic image processing apparatus according to the present embodiment shown in FIG. 9 will be described with reference to FIG. FIG. 10 is a modification of the stereoscopic image processing apparatus shown in FIG. Hereinafter, differences from the stereoscopic image processing apparatus shown in FIG. 9 will be described.

  As shown in FIG. 10, the stereoscopic image processing device is different from the stereoscopic image processing device shown in FIG. 9 in that one imaging device 100 is provided. Furthermore, the imaging apparatus 100 is different in that an optical adapter 105 is provided.

  Therefore, the imaging apparatus 100 including the optical adapter 105 captures the left-eye viewpoint image and the right-eye viewpoint image in one shooting (shutter), and generates combined image data including the two viewpoint images. can do.

(Imaging device 100)
Next, the imaging apparatus 100 according to the present embodiment will be described. The imaging apparatus 100 includes at least a lens 103 and an imaging element 130, and an optical adapter 105 can be further attached to the lens 103. As illustrated in FIG. 9, the imaging device 100 may be provided in the imaging unit 101, and the imaging device 100 may include a combining unit 131.

  Further, although the image encoding unit 132, the image control information generation unit 133, the data multiplexing unit 134, and the recording medium 135 according to the present embodiment have been described as an example, the imaging apparatus 100 is not provided. For example, the imaging apparatus 100 includes any one or any combination of the image encoding unit 132, the image control information generation unit 133, the data multiplexing unit 134, the recording medium 135, and the like. However, it can be implemented.

(Computer device 150)
A computer apparatus 150 shown in FIG. 5 includes a data separation unit 136, an image decoding unit 137, an image separation unit 138, an image conversion unit 139, and a stereoscopic display unit 140 that are included in the stereoscopic image processing apparatus shown in FIG. The computer device 150 may further include an image encoding unit 132, an image control information generation unit 133, a data multiplexing lower part 134, and a recording medium 135.

  The computer device 150 takes in the combined image data photographed by the imaging device 100 with the optical adapter 105 shown in FIG. Alternatively, the computer device 150 captures image data captured without the optical adapter 105 attached. Examples of the imaging apparatus 100 according to the present embodiment include a digital still camera and a digital video camera.

  Further, the computer device 150 generates stereoscopic image data from the captured combined image data or two pieces of image data that are continuously captured. The generated stereoscopic image data is displayed on the stereoscopic display unit 140 as a stereoscopic image.

  Note that image control information generated by the image control information generation unit 133 is added to image data such as a combined image captured from the imaging apparatus 100 to the computer apparatus 150.

(Stereoscopic image file)
Next, the stereoscopic image file according to the present embodiment will be described with reference to FIG. FIG. 11 is an explanatory diagram showing an outline of the data structure of the stereoscopic image file according to the present embodiment.

  As shown in FIG. 11, the stereoscopic image file is a JPEG format compressed data file having an extension of “.jpg”, such as a file name “file 1.jpg”.

  In addition, the stereoscopic image file is recorded in accordance with a DCF (Design rule for Camera File system) standard, and an application marker segment (APP1) is inserted therein.

  APP1 is arranged immediately after an SOI (Start Of Image) marker indicating the start of a stereoscopic image file.

  Further, immediately after APP1, combined image data is arranged, and finally an EOI (End Of Image) indicating the end of the stereoscopic image file is arranged. The combined image will be described later.

  As shown in FIG. 11, the APP1 area includes identification information of Exif (Exchangeable image file format) and an attached information main body (Tiff header, IFD 0, IFD 1). The size of APP1 including all of these must not exceed 64 kBytes according to the JPEG standard.

  The attached information has a Tiff structure including a file header (Tiff header), and can record a maximum of two IFDs (IFD0 (0th IFD) and IFD1 (1st IFD)). Note that IFD is an abbreviation of “Image File Directory”.

  The IFD0 records attached information related to a compressed image (main image) or a stereoscopic image (3D image). As shown in FIG. 11, in the IFD0 area, an Exif pointer section into which an Exif IFD pointer enters, a GPS pointer section into which a GPS IFD pointer enters, and a 3D pointer section into which a 3D IFD pointer enters.

  In the IFD0 area, an Exif IFD, an Exif IFD Value, a GPS IFD, a GPS IFD Value, a 3D IFD, and a 3D IFD Value are arranged after each pointer portion.

  In the Exif IFD and the Exif IFD Value, tags or tag values related to image data characteristics, structure, user information, shooting conditions, or date / time are recorded. For example, a tag “UserComment” tag for a user comment, a tag “ExposureTime” for indicating an exposure time, a tag “Flash” tag for indicating the presence or absence of a flash, and the like are exemplified.

  The GPS IFD and the GPS IFD Value are areas in which tags or tag values related to GPS (global positioning system) are recorded. For example, a tag “GPSLatitude” indicating latitude, a tag “GPSAltitude” indicating altitude, and the like are exemplified.

  In the 3D IFD and 3D IFD Value, a tag or tag value (Value) related to image control information for controlling processing for converting to a stereoscopic image that is a 3D image is recorded. The image control information will be described in detail later. Also, the combined image file according to the present embodiment has almost the same data structure as the stereoscopic image file.

  Note that the data structure of the stereoscopic image file according to the present embodiment is not limited to such an example, and the data structure can be implemented even if it is another data structure. For example, when the data structure of the stereoscopic image file is composed of JPEG data, each JPEG data may include a JPEG header, compressed image data, and EOI (End Of Image). The JPEG header includes attached information such as SOI (Start Of Image) and color management information. Further, viewpoint information such as the viewpoint number of the viewpoint image data may be present in each JPEG header, for example. A plurality of viewpoint images having different viewpoints can be file end information indicating a file header, a plurality of subsequent JPG data, and the end of the whole. When the header for the entire image does not exist (in the case of a combined image), the viewpoint information is stored in the JPEG header (application marker / IFD).

(Image information)
The stereoscopic image file according to the present embodiment includes at least viewpoint image data serving as a main image generated by shooting or recording, and image control information. The viewpoint image data and the image control information are defined in the DCF standard.

(Image format)
Next, the combined image data according to the present embodiment will be described with reference to FIG. FIG. 12 is an explanatory diagram showing a schematic configuration of combined image data according to the present embodiment.

  As shown in FIGS. 12A and 12B, the parallax image data according to the present embodiment is obtained from a left eye viewpoint image (L image) and a right eye viewpoint image (R image). It is configured. Note that the L image and the R image are viewpoint images captured at each viewpoint. The viewpoint image, which is a still image, is synthesized into a 3D stereoscopic image.

  The combined image data is image data configured such that L image data and R image data are integrated into one image. Therefore, compared to the parallax image data, the combined image data is common in that it is composed of a plurality of viewpoint images, but the parallax image data is largely different from the parallax image data particularly in that each viewpoint image is not separate or independent. Different. The L image and the R image of the combined image are combined and integrated.

  A combined image 350-1 shown in FIG. 12A is configured such that an L image and an R image are arranged side by side in the horizontal direction and on the left and right. Also, in the combined image 350-2 shown in FIG. 12B, the L image and the R image are integrally combined vertically in the vertical direction.

  The combined image according to the present embodiment has been described by taking the case of two viewpoints as an example. However, the combined image is not limited to such an example, and images captured from a plurality of viewpoints are combined and recorded into a single combined image. Even if it is, it can be implemented.

  Further, as shown in FIG. 12C, in addition to the combined image 350-2 combined vertically in the vertical direction, a total of four combined images (350-3, 350-4, 350-5) are combined. , 350-6).

  A combined image (350-3, 350-4, 350-5, 350-6) shown in FIG. 12C is obtained by converting the upper and lower images from the arrangement form of the combined image 350-2 combined vertically in the vertical direction. In an integrated state, it is arranged by tilting it to the left or right (90 ° clockwise rotation or 90 ° clockwise rotation).

  The combined image (350-3, 350-4, 350-5, 350-6) is displayed on the display having an aspect ratio of 3: 4, in particular, than the combined image 350-2 shown in FIG. When displaying, it fits well and can be displayed in one screen.

  Further, the combined image (350-3, 350-4, 350-5, 350-6) is recorded on the recording medium 135 together with image control information for synthesizing the stereoscopic image. Thus, it is possible to easily distribute the combined image and the image control information.

  Note that the recording method is not limited as long as the combined image (350-3, 350-4, 350-5, 350-6) and the image control information are linked to each other. For example, the present invention can be implemented even when the combined image and the image control information are recorded in the same file, or when the combined image and the image control information are separately recorded on different servers connected to the network.

  In addition, by compressing the combined image (350-3, 350-4, 350-5, 350-6) in the horizontal direction, the combined image is combined with the stereoscopic image and displayed when rotated. Since only the vertical component of the image in the restored state is compressed, a stereoscopic image can be displayed without degrading the image quality in the horizontal direction.

  Of the combined images (350-3, 350-4, 350-5, 350-6), there are two types of designated patterns as to which arrangement pattern is stored. The first pattern is an ID such as “1”, “2”, “3”, “4”, etc. for each of the combined images (350-3, 350-4, 350-5, 350-6). It is a pattern that designates an arrangement pattern by giving.

  In the second pattern, the combined image (350-3, 350-4, 350-5, 350-6) is first combined image (350-3, 350-5) or combined image (350-4, 350). −6) and a pattern for designating whether the right viewpoint image (R image) is arranged on the right side or the left side. Details will be described later.

  The combined image 350-3 is an image in a state where the combined image 350-2 illustrated in FIG. Furthermore, the combined image 350-3 is an image having a configuration in which an R image is arranged on the left side and an L image is arranged on the right side, as shown in FIG. In addition to the left R image and the right L image, the image orientation (top and bottom) is the horizontal left direction on the upper side and the horizontal right direction on the lower side.

  The combined image 350-4 is an image in a state where the combined image 350-2 shown in FIG. Further, as shown in FIG. 12C, the combined image 350-4 is an image having a configuration in which an L image is arranged on the left side and an R image is arranged on the right side. In addition to the left L image and the right R image, the image orientation (top and bottom) is the horizontal right direction on the upper side (ceiling side) and the horizontal left direction on the lower side (ground side).

  The combined image 350-5 is an image in which the L image and the R image are switched left and right from the state of the combined image 350-3, with the orientation of the image unchanged, and the combined image 350-6 is a combined image. From the state of 350-4, the image is in the state in which the L image and the R image are interchanged with the left and right being left unchanged.

  In the combined image (350-3, 350-4, 350-5, 350-6) shown in FIG. 12C, the image frames of the R image and the L image that are the respective viewpoint images, for example, 2 × vertical × horizontal Even a wide image such as 3 can be displayed side by side in one screen, and the R image and L image can be rotated by 90 degrees to display the combined stereoscopic image in a wide image. Is done. Hereinafter, unless otherwise specified, the combined image indicates any one of the images shown in FIG.

(Multi-viewpoint images)
Here, with reference to FIG. 13, a viewpoint image in the case of imaging from a plurality of viewpoints according to the present embodiment will be described. FIG. 13 is an explanatory diagram illustrating a schematic configuration of a viewpoint image when captured from a plurality of viewpoints according to the present embodiment.

  As shown in FIG. 13A, a plurality of imaging devices 100 (100-1, 100-2, 100-3,..., 100-6) are arranged for the subject 230. In order to identify the imaging device 100, a viewpoint position is attached to each imaging device 100.

  Note that the imaging apparatus 100 (100-1, 100-2, 100-3,..., 100-6) according to the present embodiment is taken as an example when it is arranged for shooting from six viewpoints. As will be described, the present invention is not limited to this example as long as one or more imaging devices 100 are arranged.

As shown in FIG. 13A, the viewpoint position is based on the (x, y) coordinates in the horizontal direction (x) and the vertical direction (y) in order from the upper left, which is the imaging device 100-1, A ₁ , A ₂ , A ₃ ,... For example, when A ₁ is expressed by coordinates, (horizontal direction coordinates, vertical direction coordinates) = (1, 0).

  Therefore, by adding the viewpoint position of each viewpoint to the viewpoint image captured from the imaging device 100, it is possible to identify from which viewpoint the viewpoint image is captured.

For example, when the viewpoint position of the imaging apparatus 100 is “A ₁ ”, the viewpoint position of the viewpoint image captured by the imaging apparatus 100 is also “A ₁ ” as illustrated in FIG.

Therefore, the image control information generation unit 133 of the imaging apparatus 100 generates image control information, and sets the viewpoint position “A ₁ ” (1, 0) in a field of a predetermined tag in the image control information. The predetermined tag relating to the viewpoint position will be described later.

(Image control information)
In order to synthesize and display stereoscopic image data from the combined image data according to the present embodiment, it is necessary to record image control information as tag information on, for example, a hard disk or a recording medium as described above. is there.

  Next, tags related to image control information according to the present embodiment will be described. The tag according to the present embodiment includes a “3D tag” indicating a 3D image, an “image control information size tag” indicating the size of the image control information, and a “3D image size tag” indicating the size of the 3D image. "User data area tag (ApplicationData)" that allows the user to manage arbitrary data, "Data configuration type tag" for managing the data configuration type, or "Image control information indicating the version of the image control information" Version tag "exists. By using a tag included in the image control information, an appropriate stereoscopic image can be generated, edited, and subjected to various processes.

Further, the tags related to the image control information according to the present embodiment include, for example, tags (viewpoint position tags) for indicating viewpoint positions “A ₁ ” to “B ₃ ” shown in FIG. 13 and 3D images. “Combined image tag” indicating whether the image is a combined image, “Rotation tag” indicating the rotation status of the 3D image, etc., “Cutout size” indicating the horizontal size of a cut-out area cut out from the 3D image "Tag X (CropSizeX)", "Cut size tag Y (CropSizeY)" indicating the vertical size of a cut-out area cut out from the 3D image, and "Horizontal offset of the cut-out area in the 3D image""Cutoff offset tag X (CropOffsetX)" or "Cutoff offset tag Y (CropOffsetY)" indicating the vertical offset of the cutout region in the 3D image ”Exists. Note that “CropOffsetX” and “CropOffsetY” indicate the offset of each viewpoint when a 3D image is formed from a plurality of viewpoints. The rotation tag is set with, for example, an angle at which the viewpoint image is rotated, such as rotation angle information.

  Further, the tag relating to the image control information according to the present embodiment includes “effective size tag X (ValidSizeX)” indicating the horizontal size of the effective area in the 3D image, and the vertical size of the effective area in the 3D image. "Effective size tag Y (ValidSizeY)" indicating "effective offset tag X (ValidOffsetX)" indicating the horizontal offset of the effective area in the 3D image, or "effective offset indicating the vertical offset of the effective area in the 3D image" A tag Y (ValidOffsetY) "exists. Note that “effective offset tag X” and “effective offset tag Y” indicate the offset of each viewpoint when a 3D image is formed from a plurality of viewpoints.

  Furthermore, the tag relating to the image control information according to the present embodiment is a tag for displaying a stereoscopic image of an appropriate size on the stereoscopic display unit 140 capable of displaying a 3D image (stereoscopic image). There is a certain “assumed display size tag” or a “remark point tag” for indicating a target reference point of a stereoscopic image.

  For example, the “rotation tag” consists of 3 bytes. The “rotation tag” indicates the rotation angle of each viewpoint image in the combined image, the presence / absence of left / right mirror inversion, and a value of 0 to 7 is set. The rotation angle and presence / absence of the mirror are determined according to the values of 0 to 7.

  “CropOffsetX” is a tag consisting of 6 bytes for each viewpoint. Therefore, for example, in the case of two viewpoints, it is 12 bytes. “CropOffsetX” indicates the horizontal offset of the cut-out area. The horizontal offset of the cutout area indicates the start position (start point) of the horizontal cutout area.

  “CropOffsetY” is a tag consisting of 6 bytes for each viewpoint. Therefore, for example, in the case of two viewpoints, it is 12 bytes. “CropOffsetY” indicates a vertical offset of the cutout area. Including the vertical shift correction amount of the image of each viewpoint, and when the shift amount is unknown, the value “0” is set in the “CropOffsetY”. The vertical offset of the cutout area indicates the start position (start point) of the vertical cutout area. By determining the vertical / horizontal offset of the cutout area, the start position of the cutout area is determined.

  “ValidSizeX” is a 6-byte tag for each viewpoint. Note that “ValidSizeX” is the horizontal size of an effective area indicating an area that can be converted from a combined image to a 3D image.

  “ValidSizeY” is a 6-byte tag for each viewpoint. “ValidSizeY” is the vertical size of the effective area.

  “ValidOffsetX” is a 6-byte tag for each viewpoint. For example, in the case of 3 viewpoints, it is 18 bytes. Note that “ValidOffsetX” is an offset in the horizontal direction of the effective area. The horizontal offset of the effective area indicates the start position (start point) of the effective area in the horizontal direction.

  “ValidOffsetY” is a 6-byte tag for each viewpoint. For example, in the case of 3 viewpoints, it is 18 bytes. “ValidOffsetY” is an offset in the vertical direction of the effective area. The vertical offset of the effective area indicates the start position (start point) of the effective area in the vertical direction.

  The “assumed display size tag” is a 4-byte tag. For example, the type and size of the stereoscopic display unit 140 are set in the tag field of “assumed display size tag”. The size of the stereoscopic display unit 140 is expressed by inches in the present embodiment, but is not limited to such an example.

  The “remark point tag” is a 6-byte tag. Note that “Remarkpoint” is a tag for indicating a point of interest in a stereoscopic image.

  When displaying a stereoscopic image on the stereoscopic display unit 140, a stereoscopic image corresponding to the reference attention point is displayed at the center point of the stereoscopic display unit 140. Therefore, the center point of the stereoscopic display unit 140 matches the attention point of the stereoscopic image.

  By setting the point of interest from the “remark point” tag, for example, when the stereoscopic display unit 140 is smaller than the size of the stereoscopic image, the entire stereoscopic image cannot be displayed, but at least the stereoscopic image is displayed. It becomes possible to partially visually recognize.

(Image control information in metafile format)
Next, when the image control information generation unit 133 according to the present embodiment generates auxiliary information that is a metafile such as an XML (extensible Markup Language) format, the recording medium 135 directly multiplexes the auxiliary information. The image file may be recorded as a separate file.

  Next, various functions in the stereoscopic image processing apparatus according to the present embodiment will be described.

(Arrangement of viewpoint images in combined images)
First, the arrangement of viewpoint images in the present embodiment will be described with reference to FIG. FIG. 14 is an explanatory diagram showing a schematic configuration of the viewpoint image arrangement in the present embodiment.

(Parallel arrangement)
There are two arrangements of viewpoint images. As shown in FIG. 14A, a parallel arrangement is used in a general arrangement in which an image for the right eye (hereinafter referred to as an R image) is on the right side and an image for the left eye (hereinafter referred to as an L image) is on the left side. And Therefore, the combined image of the parallel arrangement is the same arrangement as the viewpoint seen from the left and right eyes of the user. In addition, when the combined image by the parallel arrangement is viewed by a so-called parallel method, the combined image can be viewed stereoscopically. The parallel method is a method of looking at an object in front while keeping the eye focused at a position far from the object.

(Intersection arrangement)
Next, as shown in FIG. 14 (b), when the L image is on the right side and the R image is on the left side, that is, when the inversion layer is generated and the arrangement is reversed left and right compared to the parallel arrangement, To do. In addition, in the cross arrangement or the parallel arrangement, it is specified in the combined image tag whether the right viewpoint image (R image) is the left or right side, thereby specifying the cross arrangement or the parallel arrangement. Can do.

  The parallax image with the crossing arrangement is a parallax image that can be stereoscopically viewed with the naked eye by a so-called crossing method. The parallax image generated by imaging of the imaging apparatus 100 equipped with the optical adapter 105 according to the present embodiment is a parallax image in an intersection arrangement. The intersection method is a method of viewing an object after focusing on the eyes before the object.

  When converting from a combined image to a stereoscopic image, it is possible to convert to a normal stereoscopic image because the right side is an R image because it is assumed that the image is a parallel image. In this case, since the right side is an L image, it cannot be converted into a normal stereoscopic image. Therefore, in the case of an intersection arrangement, it is necessary to specify that the left side is an R image. Note that the parallel arrangement and the cross arrangement of the parallax images can be converted by the above-described combined image tag, but details will be described later.

(Viewpoint image processing method)
Next, the viewpoint image processing method according to the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing an outline of the viewpoint image processing method according to the present embodiment.

  First, it is set in advance in the imaging apparatus 100 whether to generate combined image data in parallel arrangement or combined image data in cross arrangement. When setting, the setting is made on a GUI screen or the like displayed on a display unit (not shown) of the imaging apparatus 100. In addition, when only one of the parallel arrangement and the cross arrangement is determined in advance, it is not necessary to set.

  Next, when a subject is imaged by the imaging device 100, the synthesis unit 131 of the imaging unit 101 generates combined image data from the viewpoint image data transmitted from the imaging element 130 according to the set parallel arrangement or intersection arrangement. To do.

  In addition, the image control information generation unit 133 generates image control information for the generated combined image data, and sets a value indicating the parallel arrangement or the intersection arrangement in the “combined image tag” tag field of the image control information. To do.

  The value set in the tag field of the “combined image tag” is, for example, “0” in the case of “parallel arrangement”, and “1” in the case of “cross arrangement”. Either. That is, if the right viewpoint image is on the right side, it is parallel arrangement, and if the right viewpoint image is on the left side, it is cross arrangement. The value set in the tag field of the “combined image tag” according to the present embodiment is not limited to this example, and any other value may be set.

  Next, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  Here, the computer device 150 reads the combined image data and the image control information from the recording medium 135. Next, the data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  When separating into L image data and R image data, the image separation unit 138 is based on the value set in the “combined image tag” of the image control information, and which of the decoded combined image data is the R image data. Recognize.

  When the value set in the “combined image tag” is “0”, the image separation unit 138 recognizes the right side image data as R image data, and when it is “1”, the image separation unit 138 The left and right viewpoint images are reversed so that they are arranged in an intersecting manner, and the left image data is recognized as R image data.

  The recognized R image data and L image data are transmitted to the image conversion unit 139 by the image separation unit 138. The image conversion unit 139 arranges the R image data and the L image data, and generates stereoscopic image data.

  The stereoscopic image data in which the R image data and the L image data are arranged is displayed by the stereoscopic display unit 140 (S600). From the above, the image processing of the viewpoint image according to the present embodiment is completed.

  As shown in FIG. 16, even after the stereoscopic image data is generated, the R image data and the L image data are reversed left and right from the GUI screen 220 displayed on the stereoscopic display unit 140 of the computer device 150. The stereoscopic image data can be generated again.

  As shown in FIG. 16, the GUI screen 220 includes a main area 222 where a stereoscopic image is displayed and a paint image 224. In the main area 222, a stereoscopic image of the generated stereoscopic image data is displayed.

  When the “LR replacement” button is pressed by an input unit (not shown) such as a mouse provided in the computer apparatus 150, the image separation unit 138 sets the value set in the “combined image tag” of the image control information. Invert. For example, in the case of “0” (parallel arrangement), it is updated to “1” (intersection arrangement).

  The image separation unit 138 recognizes which of the combined image data is R image data based on the value set in the “combined image tag” of the changed image control information, and recognizes the recognized L image data and R image data. Data is transmitted to the image conversion unit 139.

  The transmitted L image data and R image data are arranged by the image conversion unit 139, and stereoscopic image data is generated. The stereoscopic image data in which the L image data and the R image data are arranged is displayed on the stereoscopic display unit 140.

  For example, the image conversion unit 139 transmits parallax image data including an L image and an R image as illustrated in FIG. 14 to the stereoscopic display unit 140 as it is, and the stereoscopic display unit 140 transmits the parallax image data. It is possible to implement even when displaying. By staring at the parallax image displayed on the stereoscopic display unit 140 by the parallel method or the intersection method, a stereoscopic image having a different stereoscopic effect can be visually recognized by each method.

(Rotation of viewpoint image, mirror inversion)
Next, rotation of each viewpoint image or mirror inversion in the present embodiment will be described with reference to FIGS. 17, 18, 19, and 20 are explanatory views showing a schematic configuration of the arrangement of viewpoint images in the present embodiment.

Note that the viewpoint images shown in FIGS. 17 to 20 are _two -viewpoint images each having A ₁ = (0, 0) and A ₂ = (0, 1). The present invention is not limited to this, and the present invention can be implemented even in the case of parallel arrangement or the like with images of a plurality of viewpoints (A ₁ , A ₂ ,...).

(Rotation angle 0 degree)
As shown in FIG. 17 (a), when the viewpoint image of each viewpoint has a rotation angle of 0 degrees, the viewpoint images (right-eye image and left-eye image) of each viewpoint are not rotated. Accordingly, the viewpoint images of the combined images before and after the rotation are the same without changing.

(Rotation angle right 90 degrees)
Next, as shown in FIG. 17B, when the viewpoint images of the respective viewpoints (A ₁ , A ₂ ) have a rotation angle of 90 degrees to the right, the viewpoint images of the respective viewpoints (right eye images, left eye images) Image) is rotated 90 degrees clockwise from the position of each viewpoint image shown in FIG.

(Rotation angle right 180 degrees)
Next, as shown in FIG. 17C, when the viewpoint images of the respective viewpoints (A ₁ , A ₂ ) have a rotation angle of 180 degrees to the right, the respective viewpoint images (right-eye image, left-eye image) Is rotated 180 degrees clockwise from the position of each viewpoint image shown in FIG.

(Rotation angle right 270 degrees)
Next, as shown in FIG. 17 (d), each viewpoint (A _{1, A 2)} when both viewpoint image rotation angle right 270 degrees, each viewpoint image (image for the right eye, the image for the left eye) Is rotated 270 degrees clockwise from the position of each viewpoint image shown in FIG.

  From the above, even when the viewpoint image of each viewpoint is rotated to an appropriate angle, all the viewpoint images are rotated by the same angle, so that even the rotated viewpoint image can be viewed stereoscopically. Images can be easily generated.

  Note that the rotation of the viewpoint image according to the present embodiment can be performed even when the viewpoint image of each viewpoint is rotated individually. Here, a case where the viewpoint image of each viewpoint is rotated will be described with reference to FIG.

(Both A ₁ and A _{2 have a} rotation angle of 0 degree)
As described above, as shown in FIG. 18 (a), when the viewpoint image of each viewpoint has a rotation angle of 0 degrees, the viewpoint images of each viewpoint (the image for the right eye and the image for the left eye) are not rotated. Therefore, the viewpoint images before and after the rotation do not change and are the same.

_{(A 2} rotation angle 90 degrees clockwise)
Next, as shown in FIG. 18 (b), only the viewpoint image _{A 2} of each viewpoint _(A 1, _{A 2)} is the case of the rotation angle 90 degrees to the right, the viewpoint image _{A 2} (for the left eye image) from the position of the viewpoint image _{a 2} shown in FIG. 18 (a), the clockwise rotation 90 degrees.

_{(A 2} rotation angle right 180 degrees)
Next, as shown in FIG. 18 (c), only the viewpoint image _{A 2} of each viewpoint _(A 1, _{A 2)} is, when the rotational angle of the right 180 °, the viewpoint image _{A 2} (for the left eye image) from the position of the viewpoint image _{a 2} shown in FIG. 18 (a), in a clockwise direction, to rotate 180 degrees.

_{(A 1} is the rotation angle 90 degrees clockwise, _{A 2} is the rotation angle right 180 degrees)
Next, as shown in FIG. 18D, among the viewpoints (A ₁ , A ₂ ), the viewpoint image of A ₁ has a rotation angle of 90 degrees to the right, and the viewpoint image of A ₂ has a rotation angle of 180 degrees to the right. If, respective viewpoint images (image for the right eye, the image for the left eye) from the position of the viewpoint images a ₁ and a ₂ shown in FIG. 18 (a), the viewpoint image a ₁ is clockwise, It rotated 90 degrees, the viewpoint image _{a 2} is clockwise rotated 180 degrees.

  Next, rotation and mirror inversion according to the present embodiment will be described with reference to FIG.

(Rotation angle 0 degree and mirror inversion)
As shown in FIG. 19A, when the viewpoint image of each viewpoint is rotated and the mirror inversion angle is 0 degree, the viewpoint images of each viewpoint (the image for the right eye and the image for the left eye) are not rotated. The mirror is inverted with respect to the vertical axis in the vertical direction where the image for the right eye and the image for the left eye are in contact with each other. The mirror inversion according to the present embodiment is performed even when one of the viewpoint images (right-eye image and left-eye image) of each viewpoint is mirror-inverted with respect to the vertical axis. It can be implemented. Details will be described later.

(Rotation angle right 90 degrees and mirror inversion)
Next, as shown in FIG. 19B, in the case where the viewpoint image of each viewpoint has a rotation angle of 90 degrees to the right and mirror inversion, each viewpoint image (right eye image, left eye image) is shown in FIG. From the position of each viewpoint image shown in (a), it is first rotated 90 degrees in the clockwise direction. Further, from the state in which each viewpoint image is rotated 90 degrees to the right, the right-eye image and the left-eye image are mirror-inverted with respect to the vertical axis in the vertical direction.

(Rotation angle right 180 degrees and mirror inversion)
Next, as shown in FIG. 19C, in the case where the viewpoint image of each viewpoint has a rotation angle of right 180 degrees and mirror inversion, each viewpoint image (right eye image, left eye image) is shown in FIG. From the position of each viewpoint image shown in (a), it is first rotated 180 degrees in the clockwise direction. Further, from the state where each viewpoint image is rotated 180 degrees to the right, the right-eye image and the left-eye image are mirror-inverted with respect to the vertical axis in the vertical direction.

(Rotation angle right 270 degrees and mirror inversion)
Next, as shown in FIG. 19D, in the case where the viewpoint image of each viewpoint has a rotation angle of 270 degrees to the right and mirror inversion, each viewpoint image (right eye image, left eye image) is shown in FIG. From the position of each viewpoint image shown in (a), it is first rotated 270 degrees in the clockwise direction. Further, from the state in which each viewpoint image is rotated 270 degrees to the right, the right-eye image and the left-eye image are mirror-inverted with respect to the vertical axis in the vertical direction.

  From the above, even if the viewpoint image of each viewpoint is rotated and mirror-reversed to an appropriate angle, since each viewpoint image is rotated and mirror-inverted by the same angle, the viewpoint image of each viewpoint is stereoscopically viewed. It is possible to easily generate a visually observable stereoscopic image.

  Note that the viewpoint image rotation and mirror inversion according to the present embodiment may be performed by rotating and mirror-inverting the viewpoint image of each viewpoint individually. Here, the case where the viewpoint image of each viewpoint is rotated and mirror-reversed will be described with reference to FIG.

(Rotation angle 0 degree)
As shown in FIG. 20 (a), as described above, when the viewpoint image of each viewpoint has a rotation angle of 0 degree and no mirror inversion, the viewpoint images of each viewpoint (images for the right eye and images for the left eye). Neither rotated nor mirror reversed. Therefore, the viewpoint images before and after rotation and mirror inversion are the same without changing.

_{(A 2} rotation angle 90 degrees clockwise and mirrored)
Next, as shown in FIG. 20 (b), only the viewpoint image _{A 2} of each viewpoint _(A 1, _{A 2)} is the case of the rotation angle 90 degrees to the right and mirrored, the viewpoint image _{A 2} (Left image for the eye) from the position of the viewpoint image a ₂ shown in FIG. 20 (a), first, clockwise, to rotate 90 degrees. Furthermore, from the state where the viewpoint image A ₂ is rotated clockwise 90 degrees, with respect to the vertical direction of the vertical axis, only the viewpoint image A ₂ is, to mirrored.

_{(A 2} rotation angle right 180 degrees and the mirror inversion)
Next, as shown in FIG. 20 (c), only the viewpoint image _{A 2} of each viewpoint _(A 1, _{A 2)} is, when the rotational angle of the right 180 ° and mirrored, the viewpoint image _{A 2} (Left image for the eye) from the position of the viewpoint image _{a 2} shown in FIG. 20 (a), first, clockwise, to rotate 180 degrees. Furthermore, from the state where the viewpoint image A ₂ is rotated right 180 degrees, with respect to the vertical direction of the vertical axis, only the viewpoint image A ₂ is, to mirrored.

_{(A 1} is the rotation angle 90 degrees clockwise and mirrored, _{A 2} is the rotation angle right 180 degrees and the mirror inversion)
Next, as shown in FIG. 20D, among the viewpoints (A ₁ , A ₂ ), the viewpoint image of A ₁ has a rotation angle of 90 degrees right and mirror inversion, and the viewpoint image of A ₂ has a rotation angle of right for 180 and mirrored, (image for the right eye, the image for the left eye) each viewpoint image, the position of the viewpoint images a ₁ and a ₂ shown in FIG. 20 (a), a ₁ viewpoint image is clockwise rotated 90 degrees, the viewpoint image _{a 2} is clockwise rotated 180 degrees. Furthermore, from the state where the viewpoint image A ₁ is rotated clockwise 90 degrees, with respect to the vertical direction of the vertical axis, from a state where the viewpoint image A ₁ is mirrored, the viewpoint image A ₂ is rotated right 180 degrees, vertical directions with respect to a vertical axis, the viewpoint image a ₂ is mirrored.

  Next, the combined image recording process according to the present embodiment will be described with reference to FIG. FIG. 21 is a flowchart showing an outline of the combined image recording process according to the present embodiment.

  As shown in FIG. 21, first, a subject is photographed by the imaging device 100 (S650). The synthesizing unit 131 of the imaging unit 101 converts the viewpoint image data transmitted from the imaging element 130 immediately after shooting into combined image data according to environment information set in the imaging device 100 in advance. For example, the image data may not be converted into combined image data, but simply converted into viewpoint image data as shown in FIG.

  Next, in order to convert into combined image data, the composition unit 131 cuts out only a necessary image area from the viewpoint image data immediately after imaging (S651). Although the clipping process will be described later, the necessary image area can be exemplified when the effective area is determined in advance by the characteristics of the adapter or the like, or when the area is set in the environment information in advance.

  Next, when necessary image regions are cut out from the left and right viewpoint images (S651), as described above, the arrangement pattern of the combined image data is set (S652). For example, when the arrangement pattern is set as environment information via a display unit (not shown) on which the GUI of the imaging apparatus 100 is displayed in advance, the arrangement pattern is set every time shooting is performed. Etc. can be illustrated.

  Here, setting of the arrangement pattern will be described with reference to FIG. As shown in FIG. 22, the environment information setting screen for converting into combined image data includes a designation area 654 for designating which side the right viewpoint image is arranged on the left and right sides, and both viewpoint images are tilted in which direction. And a designated area 655 for designating whether or not the current state has been reached. FIG. 22 is an explanatory diagram showing a schematic configuration of the environment information setting screen according to the present embodiment.

  By selecting a radio button displayed in each of the designated area 654 and the designated area 655 with an input unit (not shown), the arrangement pattern of the combined image data is set. Note that the environment information setting screen according to the present embodiment has been described by taking an example in which the environment information setting screen is displayed on the display unit of the imaging apparatus 100. However, the environment information setting screen is not limited to such an example. It may be displayed on the device 150 or the like. In the above case, when the “rotate” button shown in FIG. 16 is pressed, the environment information setting screen is displayed and converted into combined image data. Further, the combined image according to the present embodiment has two viewpoints, but is not limited to such an example, and may be a plurality of viewpoints. In addition to the above radio buttons, if the mark can be selected and selected, the present invention is not limited to this example. For example, a check box may be used.

  After the arrangement pattern setting (S652), the composition unit 131 rotates the L image data (left eye image) and the R image data (right eye image) of each viewpoint image according to the set arrangement pattern. , Integrate and generate combined image data. Note that in the combined image data, the L image data and the R image data are integrated in any arrangement form of the combined image 350 shown in FIG.

  Further, the image control information generation unit 133 generates image control information for the generated combined image data, and sets a value indicating the type of mirror inversion and rotation in the tag field of the “rotation tag” of the image control information. To do. The “rotation tag” will be described later.

  Next, when the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135 (S653), a series of processing of the combined image recording method according to the present embodiment is performed. finish.

  By recording and managing the combined image and the image control information of the combined image on the recording medium 135, it is possible to easily generate and display a stereoscopic image from the combined image and the image control information. . Furthermore, the combined image and the image control information can be distributed via the network, and the stereoscopic image can be displayed on another computer device 150 or the like.

  Next, a stereoscopic image display process for displaying a stereoscopic image according to the present embodiment will be described with reference to FIG. FIG. 23 is a flowchart showing an outline of the stereoscopic image display processing according to the present embodiment.

  As shown in FIG. 23, first, the computer device 150 reads the combined image data and the image control information recorded on the recording medium 135 in accordance with an instruction from an input unit (not shown).

  Next, the data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  Next, as shown in FIG. 23, the image separation unit 138 first cuts out a necessary image area before separating the decoded combined image data into L image data and R image data (S656). The image separation unit 138 recognizes which side of the combined image is the right viewpoint image based on the value of the combined image tag set in the image control information.

  Next, as shown in FIG. 23, for display on the stereoscopic display unit 140, the image separation unit 138 counts from the 2nd byte to 3 bytes from the least significant digit of the “rotation tag” field of the image control information. Based on the value set to the eyes (“1” or “3”), the entire viewpoint image data (L image data, R image data) is integrated as a whole, and the entire rotation angle is 90 degrees to the right. The entire rotation is made 270 degrees to the right (S657). The entire rotation is not to rotate each viewpoint image data individually but to rotate the entire combined image data including each viewpoint image data as a set. Therefore, the horizontally long L image data and the R image data overlap vertically. If the angle rotated during the combined image recording process is 90 degrees to the right, the rotation angle is 270 degrees to the right. If the angle rotated during the combined image recording process is 270 degrees to the right, the rotation angle is right. 90 degrees.

  For example, in the case of the combined image 350-3 among the combined images shown in FIG. 12C, by rotating the entire combined image 350-3 90 degrees clockwise, the combined image 350-3 is The L image is placed.

  In addition to the overall rotation described above, for example, the viewpoint image data may be rotated 90 degrees or 270 degrees to the right, and then array processing may be performed so that both viewpoint image data are positioned vertically.

  When the rotation process (S657) is performed, the image separation unit 138 separates the R image data and the L image data included in the combined image data, and the image conversion unit 139 separates the L image data and the R image data, respectively. Transmit to.

  As described above, the image conversion unit 139 appropriately combines the R image data and L image data transmitted from the image separation unit 138 with each other to synthesize the stereoscopic image data (S658). The image conversion unit 139 recognizes which side of the combined image is the right viewpoint image based on the value of the combined image tag set in the image control information.

  Therefore, the stereoscopic image data having a ratio that matches the aspect of the stereoscopic display unit 140 is displayed on the stereoscopic display unit 140 (S659).

  As shown in FIG. 16, even after the stereoscopic image data is generated, the R image data and the L image data are rotated or rotated from the GUI screen 220 displayed on the stereoscopic display unit 140 of the computer apparatus 150. Then, the mirror image can be reversed to generate stereoscopic image data again.

  As shown in FIG. 16, when a “rotate” button on the GUI screen 220 is pressed by an input unit (not shown) such as a mouse provided in the computer device 150 and a predetermined rotation angle or mirror inversion is selected, The image separation unit 138 changes the value set in the “rotation tag” field of the image control information. For example, the value is updated from “0” (rotation angle 0 degree) to “5” (rotation angle right 90 degrees and mirror inversion).

  Furthermore, as shown in FIG. 22, when a predetermined item is input to the displayed environment information screen when the “rotate” button is pressed, the arrangement pattern of the combined image can be changed.

  Next, a stereoscopic image display method according to the present embodiment will be described with reference to FIG. FIG. 24 is a flowchart showing an outline of a stereoscopic image display process according to the present embodiment.

  First, a type for rotating each viewpoint image data is selected in advance in the imaging apparatus 100. When selecting, it sets with the GUI screen etc. which are displayed on the display part (not shown) of the imaging device 100. FIG.

  For example, on the GUI screen, for example, “0 (rotation 0 degree)”, “1 (rotation right 90 degrees)”, “2 (rotation right 180 degrees)”, “3 (rotation right 270 degrees) for each viewpoint image. "4" (rotation angle right 90 degrees and mirror inversion) "" 6 "(rotation angle right 180 degrees and mirror inversion)" "7" (rotation angle right 270 degrees) And mirror inversion) ”or the like. If the rotation angle is set in advance and cannot be selected, it is not necessary to select it.

  Next, as illustrated in FIG. 24, when the subject is imaged by the imaging device 100, the synthesis unit 131 of the imaging unit 101 converts the viewpoint image data transmitted from the imaging element 130 to the parallel image set by the combined image tag. Arrange according to the arrangement or crossing arrangement.

  Further, in order to efficiently generate a stereoscopic image from the viewpoint image data stored in the recording medium 135, the synthesis unit 131 performs a first rotation step of rotating the viewpoint image data by 90 degrees or 270 degrees clockwise ( S601).

  After the first rotation step (S601), the synthesizer 131 integrates the L image data (left eye image) and the R image data (right eye image) of each viewpoint image, and converts the first combined image data. Generate. In the first combined image data, L image data and R image data are integrated and recorded on the recording medium 135 together with image control information.

  Further, the image control information generation unit 133 generates image control information for the generated combined image data, and sets a value indicating the type of mirror inversion and rotation in the tag field of “rotation tag” of the image control information. To do.

  Here, the values set in the tag field of “Rotation tag” will be described with reference to FIG. FIG. 25 is an explanatory diagram showing an outline of values set in the tag field of the “rotation tag” according to the present embodiment.

The “rotation tag” can be set for each viewpoint image, and the number of bytes of the tag of the “rotation tag” is 5 bytes. Of the above 5 bytes, one byte from the least significant digit to the first byte has a horizontal viewpoint position (horizontal coordinate) among the viewpoint positions (A ₁ , A ₂ , A ₃ ,...). The value to be expressed is set, and then the vertical viewpoint of the viewpoint positions (A ₁ , A ₂ , A ₃ ,...) Is included in 1 byte from the 1st byte to the 2nd byte, counting from the least significant digit. A value representing the position (vertical coordinate) is set, and a value representing the type of rotation, rotation, and mirror inversion is set in one byte from the second byte to the third byte counted from the least significant digit. The configuration is not limited to the above byte.

  As shown in FIG. 25, the value set in one byte from the second byte to the third byte counted from the least significant digit is composed of seven types of values. The bit string corresponding to the 1 byte continues from the 1st bit to the 8th bit in order. The initial setting value is “0”. Note that the values representing the types of rotation or rotation and mirror inversion according to the present embodiment have been described by taking the case of seven types of values as an example, but in the case of consisting of one or more types of values. If there is, it is not limited to such an example.

  As shown in FIG. 25, a value of “0” or “1” is set in the first bit and the second bit to indicate the value of the type of rotation. For example, when “0” is set in the first bit and “1” is set in the second bit, the value of the rotation type is “2”, and the rotation angle is 180 degrees to the right.

  Next, the third bit indicates the presence or absence of mirror inversion. When “0” is set in the third bit, mirror inversion is “none”, and when “1” is set, mirror inversion is “present”.

When the least significant digit to the first byte (horizontal viewpoint position) and the first byte to the second byte (vertical viewpoint position) are both “−1” (−1, −1), It is assumed that the rotation type value set from the second byte to the third byte is valid for all viewpoint images. For example, in the case where the viewpoint position is three viewpoints consisting of “A ₁ ”, “A ₂ ”, and “A ₃ ”, and the rotation angle is 90 degrees to the right and the mirror is inverted with respect to all viewpoint images, the “rotation tag” By setting "(5, -1, -1)" for each byte from the least significant digit, it is possible to rotate or mirror invert all three viewpoint images in one setting. . If “(−1, −1)” is not set, “(5, 0, 0)”, “(5, 0, 1)”, “(5, 0, 2)”, etc. Setting is required.

  As shown in FIG. 25, “0” is set for all of the 4th to 8th bits. However, the present invention is not limited to this example. For example, the entire viewpoint image is rotated to the 4th bit. For example, a value indicating “total rotation” may be set.

  Therefore, when rotating or rotating and mirror-inverting all the viewpoint images, the image control information generating unit 133 can efficiently set a value in the tag field of the “rotation tag” of the image control information.

  In the image display processing according to the present embodiment, the “rotation tag” has a rotation angle of 90 degrees to the right (a value “1” set in the second to third bytes of the rotation tag) or rotation. A value of 270 degrees to the right (a value “3” set in the second to third bytes of the rotation tag) is set.

  Next, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  Next, the computer device 150 reads the combined image data and the image control information from the recording medium 135.

  The data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  The image separation unit 138 separates the decoded combined image data into L image data and R image data. The separation process is exemplified by a cutting process described later. The image separation unit 138 recognizes which side of the combined image is the right viewpoint image based on the value of the combined image tag set in the image control information.

  Next, as shown in FIG. 24, in order to display on the stereoscopic display unit 140, the image separation unit 138 counts from the second byte to 3 bytes from the least significant digit of the “rotation tag” field of the image control information. From the value set to the eyes (“1” or “3”), the entire viewpoint image data (L image data, R image data) is rotated as a whole at an overall rotation angle of 90 degrees to the right or 270 degrees to the right. A second rotation process is performed (S603). The whole rotation is not to rotate each viewpoint image data individually but to rotate the entire second combined image data including each viewpoint image data as a set. Accordingly, the original state before the first rotation step (S601) is obtained, and the horizontally long L image data and the R image data overlap each other as shown in FIG.

  When the second rotation step (S603) is performed, the second combined image data including the set of the L image data and the R image data is transmitted to the image conversion unit 139 by the image separation unit 138.

  The image conversion unit 139 combines the L image data and the R image data included in the second combined image data lined up and down (S604), and generates stereoscopic image data.

  Therefore, stereoscopic image data having a ratio that matches the aspect of the stereoscopic display unit 140 is displayed on the stereoscopic display unit 140. The image conversion unit 139 recognizes which side of the combined image is the right viewpoint image based on the value of the combined image tag set in the image control information.

  As shown in FIG. 16, even after the stereoscopic image data is generated, the R image data and the L image data are rotated or rotated from the GUI screen 220 displayed on the stereoscopic display unit 140 of the computer apparatus 150. Then, the mirror image can be reversed to generate stereoscopic image data again.

  As shown in FIG. 16, when a “rotate” button on the GUI screen 220 is pressed by an input unit (not shown) such as a mouse provided in the computer device 150 and a predetermined rotation angle or mirror inversion is selected, The image separation unit 138 changes the value set in the “rotation tag” field of the image control information. For example, the value is updated from “0” (rotation angle 0 degree) to “5” (rotation angle right 90 degrees and mirror inversion).

  The image separation unit 138 rotates the second combined image data as a whole based on the value set in the “rotation tag” field of the changed image control information (S603).

  The image conversion unit 139 combines the L image data and the R image data (S604), and generates stereoscopic image data, whereby the stereoscopic image data is displayed on the stereoscopic display unit 140.

  Therefore, even when the rotation angle is 0 degrees, even when the rotation angle is recognized as 270 degrees right and converted into stereoscopic image data, appropriate stereoscopic image data can be generated. .

  A modification of the stereoscopic image display method according to the present embodiment will be described with reference to FIG. FIG. 26 is a flowchart illustrating an outline of a modified example of the image display process of the stereoscopic image according to the present embodiment. The difference from the stereoscopic image display method according to the present embodiment described above will be particularly described in detail with reference to FIG.

  First, a type for rotating each viewpoint image data is selected in advance in the imaging apparatus 100. When selecting, it sets with the GUI screen etc. which are displayed on the display part (not shown) of the imaging device 100. FIG.

  Next, as illustrated in FIG. 26, when the subject is imaged by the imaging apparatus 100, the synthesis unit 131 of the imaging unit 101 converts the viewpoint image data transmitted from the imaging element 130 to the set parallel arrangement or cross arrangement. Arrange according to.

  Further, the synthesis unit 131 performs a first rotation process of rotating in the same direction by 90 degrees or 270 degrees (S606).

  After the first rotation step (S606), the synthesis unit 131 integrates the L image data and the R image data to obtain first combined image data. In the first combined image data, L image data and R image data are integrated. The aspect ratio of the L image data and the R image data is a vertically long image.

  Further, the image control information generation unit 133 generates image control information for the generated combined image data, and a value indicating the type of rotation and mirror inversion or rotation in the tag field of “rotation tag” of the image control information. Set.

  In the image display processing according to the present embodiment, the field of the “rotation tag” includes each viewpoint position and a rotation angle right of 90 degrees (“1”) or a rotation angle of right 270 degrees (“3”). Value is set.

  Next, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  Further, the computer device 150 reads the combined image data and the image control information from the recording medium 135.

  The data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  The image separation unit 138 separates the decoded first combined image data into L image data and R image data. The separation process is exemplified by a cutting process described later.

  Next, as shown in FIG. 26, a screen rotation process is performed in which the screen of the stereoscopic display unit 140 is rotated by 90 degrees to the right or 270 degrees to the right (S608). In the stereoscopic display unit 140 before rotation, the L image data and the R image data are vertically displayed on the left and right, and by the rotation, the stereoscopic display unit 140 displays the vertical L image data and the R image data. Are displayed on the left and right.

  When the screen rotation step (S608) is performed, the L image data and the R image data are transmitted to the image conversion unit 139 by the image separation unit 138.

  The image conversion unit 139 combines the vertically long L image data and the R image data arranged side by side (S610), and generates stereoscopic image data.

  Accordingly, the stereoscopic image data having a ratio that matches the aspect of the vertically long screen of the stereoscopic display unit 140 is displayed on the stereoscopic display unit 140. The stereoscopic display unit 140 may be a projector equipped with an adapter such as a four-mirror.

  Note that the rotation of the viewpoint image according to the present embodiment has been described by taking the case of two viewpoints as an example, but is not limited to such an example, and can be performed even when there are a plurality of viewpoints.

  Therefore, the stereoscopic display unit 140 displays a horizontally wide stereoscopic image by synthesizing the R image data and the L image data only by rotating the screen of the stereoscopic display unit 140 by 90 degrees or 270 degrees. be able to.

(Adapter model number information)
Next, adapter model number information according to the present embodiment will be described. The adapter model number information according to the present embodiment is information for identifying not only the model number of the optical adapter 105 but also the optical information of the optical adapter 105, and the adapter information corresponds to the adapter model number information. The adapter model number information is recorded, for example, in a tag field such as “user data area tag” among a plurality of image control information tags.

  As shown in FIG. 2, the optical adapter 105 can be exemplified by a two-mirror adapter including a mirror 121 and a mirror 122, for example. Light incident from the viewpoint in the direction of the arrow “right” is reflected by the mirror 121 and the mirror 122, and a parallax image is generated as an R image.

  Also, the light incident from the viewpoint in the direction of the arrow “left” shown in FIG. 2 is not reflected by the mirror 121 and the mirror 122 but passes through as it is, and a parallax image is generated as an L image.

  The R image generated by being reflected by the mirror 121 and the mirror 122 has deteriorated image quality (luminance, saturation, resolution, etc.), for example, as compared with the L image. In addition, the luminance of the peripheral part (upper side part, lower side part, and left side part) of the R image is lower than that of the central part. Also, the R image is a rectangular image that is distorted into a trapezoid.

  Since the luminance, saturation, resolution, or trapezoidal distortion is unique to the optical adapter 105, a correction amount for correcting the R image is determined in advance when converting to a stereoscopic image.

  Accordingly, the computer device 150 stores a storage unit (not shown) so that correction information such as luminance, saturation, resolution, or trapezoidal distortion as optical information has a one-to-one correspondence with adapter model number information of each optical adapter 105. By storing, it can be converted into an appropriate stereoscopic image. The correction information is stored as correction content information indicating the correction content for each optical adapter 105. The correction content information will be described later.

  The optical adapter 105 according to the present embodiment is a two-mirror adapter provided with a mirror 121 and a mirror 122, but is not limited to this example, and as shown in FIG. 28, a four-mirror adapter, an angle prism Examples include an adapter, an anamorphic lens adapter, a vari-angle prism adapter (VAP adapter), and the like.

  The adapter model number information according to the present embodiment varies depending on the type of the optical adapter 105 and the like. Hereinafter, the optical adapter 105 other than the two-mirror adapter will be described with reference to FIG.

  An optical adapter 105-1 shown in FIG. 28A is a four-mirror adapter including mirrors 551-1, mirror 551-2, mirror 551-3, and mirror 551-4.

  By providing the optical adapter 105-1 in the imaging apparatus 100, it is possible to take a viewpoint image consisting of two viewpoints by one shooting (shutter). For example, two viewpoint images can be taken at a time, such as an L image for the left eye and an R image for the right eye.

  In the optical adapter 105-1 according to the present embodiment, trapezoidal distortion or the like occurs as in the case of the two-mirror adapter provided with the mirror 121 and the mirror 122 described above.

  Therefore, for example, when converting from left-eye and right-eye viewpoint images to stereoscopic images, it is necessary to perform correction such as eliminating the trapezoidal distortion of the viewpoint images in advance and then convert to stereoscopic images. Note that the correction amount for the trapezoidal distortion is unique to the optical adapter 105-1.

  The correction relating to the optical adapter 105-1 is stored as correction content information indicating the correction content for each optical adapter 105. The correction content information will be described later.

  Next, the optical adapter 105-2 shown in FIG. 28B is a chevron prism adapter. By providing the imaging apparatus 100 with the optical adapter 105-2 including the angle prism adapter, it is possible to take, for example, a two-viewpoint image or the like by one shooting (shutter) as in the case of the four-mirror adapter or the like.

  The optical adapter 105-2 according to the present embodiment will be described by taking as an example a case where it is composed of one angle prism adapter, but is not limited to such an example. Even if it is comprised from a several angle prism adapter, it can implement.

  In the optical adapter 105-2 according to the present embodiment, trapezoidal distortion, chromatic aberration, and the like are generated as in the case of the two-mirror adapter and the four-mirror adapter described above.

  Therefore, for example, when converting the left-eye and right-eye viewpoint images into stereoscopic images, the viewpoint images are corrected in advance to eliminate trapezoidal distortion and chromatic aberration, and then converted into stereoscopic images. There is a need. It should be noted that the correction amount of the trapezoidal distortion and chromatic aberration is unique to the optical adapter 105-2. Further, with regard to correction of chromatic aberration according to the present embodiment, for example, correction of blackening a portion where a color is blurred can be exemplified.

  The correction relating to the optical adapter 105-2 is stored as correction content information indicating the correction content for each optical adapter 105. The correction content information will be described later.

  An optical adapter 105-3 shown in FIG. 28C is an anamorphic lens adapter including a mirror (not shown) and an anamorphic lens. The anamorphic lens is a lens having optical axes with different focal lengths in two vertical and horizontal directions, and can optically change the aspect ratio of the image.

  Therefore, since the optical adapter 105-3 includes the anamorphic lens and a mirror (not shown), the optical adapter 105-3 is provided in the imaging apparatus 100, so that one imaging can be performed. With (Shutter), it is possible to take a viewpoint image in which the aspect ratio of two viewpoints is changed.

  The aspect ratio scaling according to the present embodiment is, for example, from 3: 4 to 9:16, but is not limited to this example, and any other magnification is acceptable as long as the magnification is less than about 2 times. Even so, it is possible to change the aspect ratio.

  In the optical adapter 105-3 according to the present embodiment, trapezoidal distortion or the like occurs as in the case of the above-described two-mirror mirror adapter, four-mirror adapter, or the like.

  Therefore, for example, when converting the left-eye and right-eye viewpoint images into stereoscopic images, it is necessary to correct the viewpoint images in advance, such as trapezoidal distortion, before converting them into stereoscopic images. Note that the correction amount of the trapezoidal distortion and the like is unique to the optical adapter 105-3.

  In addition, since the aspect ratios of the left-eye and right-eye viewpoint images are scaled, it is necessary to correct the aspect ratio when converting from the two-viewpoint images to the stereoscopic images as necessary. is there. Note that the change in aspect ratio is specific to each anamorphic lens.

  The correction related to the optical adapter 105-3 is stored as correction content information indicating the correction content for each adapter model number of the optical adapter 105. The correction content information will be described later.

  Next, a method for setting adapter model number information according to the present embodiment will be described.

  First, the type of adapter model number to be attached to the apparatus imaging apparatus 100 is selected in advance. When the adapter model number is selected, the adapter model number is set on the GUI screen 221 displayed on the display unit (not shown) of the imaging apparatus 100.

  As shown in FIG. 27, on the GUI screen 221, each adapter model name consisting of “No. 1” to “No. 6”, an adapter selection unit 223 for selecting an adapter model number, and a “select” button are displayed. Is done.

  The adapter model number “No. 1” shown in FIG. 27 is “MONY FS_Adaptor 1.0”, and the “No. 1” adapter is a four-mirror adapter. The adapter number of “No. 2” is “MONY FS_Adaptor 2.0”, and the “No. 1” adapter is an angle prism adapter.

  Further, the adapter model number “No. 3” shown in FIG. 27 is “MONY FS_Adaptor 3.0”, and the “No. 3” adapter is a two-mirror adapter. The adapter model number “No. 4” is “MONY FS_Adaptor 4.0”, and the “No. 4” adapter is an adapter provided with a four-mirror and an anamorphic lens.

  The adapter model number of “No. 5” is “MONY FS_Adaptor 5.0”, and the “No. 5” adapter is an adapter provided with an angle prism and an anamorphic lens.

  The adapter number of “No. 6” is “MONY FS_Adaptor 6.0”, and the “No. 6” adapter is an adapter provided with two mirrors and an anamorphic lens.

  Next, when the adapter model number of the optical adapter 105 attached to the imaging apparatus 100 is selected from the displayed adapter model name names by an input unit such as a jog dial, the adapter selection unit 223 displays the selected adapter. The adapter model number information in which the model number is set is generated. Note that the adapter model number information is temporarily stored in a storage unit (not shown) of the imaging apparatus 100 or the like.

  Next, when the subject is imaged by the imaging device 100, the synthesis unit 131 of the imaging unit 101 synthesizes the viewpoint image data (L image data and R image data) transmitted from the imaging device 130, and combines the combined image data. Generate.

  Further, the image control information generation unit 133 generates image control information for the generated combined image data, and temporarily stores the adapter model number information in the tag field of the “user data area tag” of the image control information. Set to.

  The image control information generation unit 133 sets the adapter field in the tag field of the “user data area size tag” before setting the adapter model number information in the tag field of the “user data area tag”. Set the data size (byte length) etc. of the model number information.

  When the adapter model name “No. 2” shown in FIG. 27 is selected, “MONY FS_Adaptor 1.0” is set in the adapter model number information. The adapter model number information is set in the tag field of “user data area tag”.

  When the adapter model number information is set in the tag field of the “user data area tag”, the tag field such as “user data area size tag” indicating the presence / absence of setting of the adapter model number information in advance, for example, “1” indicating the setting is stored. If not set, “0” is stored.

  Next, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  The computer device 150 reads the combined image data and the image control information from the recording medium 135. Next, the data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  The image separation unit 138 acquires the adapter model number information set in the “user data area tag” field of the image control information, and based on the adapter model number information, the adapter model number information from the storage unit (not shown). Correction content information (not shown) that is uniquely determined as follows.

  The correction content information is information indicating the content of correction performed for each adapter model number. For example, when the adapter model number is “MONY FS_Adaptor 1.0”, correction content information includes correction for removing trapezoidal distortion, correction for lowering the brightness level of R image data by a predetermined level, and saturation level of R image data. For example, correction for raising the level to a predetermined level. The correction content information is information unique to each adapter model number.

  The image separation unit 138 corrects each viewpoint image data (L image data or R image data) of the decoded combined image data based on the correction content information. If there is no correction content information related to the adapter model number, it is transmitted to the image conversion unit 139 without any correction. Note that the image separation unit 138 and the image conversion unit 139 recognize which is the right viewpoint image of the combined image based on the value set in the combined image tag in the image control information.

  The R image data and the L image data transmitted from the image separation unit 138 are converted into stereoscopic image data by the image conversion unit 139.

  Even after the stereoscopic image data corrected based on the correction content information is generated, the R is displayed via a GUI screen (not shown) displayed on the stereoscopic display unit 140 of the computer device 150. It is also possible to correct the image data or the L image data and generate the stereoscopic image data again.

(Paint image data, hierarchical information)
Next, the paint image data and hierarchy information according to this embodiment will be described. The paint image data according to the present embodiment is still image data including, for example, characters, figures, symbols, or frames.

  When the paint image data is a frame image, the frame image data indicates, for example, an edge of the viewpoint image or an image around a frame (frame) of the viewpoint image.

  Note that, among the paint image data according to the present embodiment, still image data composed of characters, figures, symbols, frames, or the like is either a 2D image that is a flat display or a 3D image that is a stereoscopic display. Good.

  In addition, the case where the paint image data according to the present embodiment is still image data has been described as an example. However, the present invention is not limited to this example. For example, the paint image data is moving image data such as animation. Even if it is possible.

  The hierarchical information according to the present embodiment can be set for each paint image such as a character, a figure, a symbol, or a frame. It is also possible to set hierarchical information for the entire viewpoint image.

  When superimposing the paint image data on the stereoscopic image data, first, it is necessary to set a value indicating that the paint image data exists in the tag field of the “data configuration type tag” of the image control information.

  The “data configuration type tag” indicates a data type generated by the combining unit 131 of the imaging apparatus 100. For each data type, it indicates whether or not the data exists in the stereoscopic image file. The data type includes, for example, parallax image data (combined image data) or paint image data. The paint image data may exist so as to be linked to the outside of the stereoscopic image file.

  The tag field of “data structure type tag” is 1 byte, and the least significant bit (bit 0) of the 1 byte (8 bits) indicates the presence / absence of parallax image data or combined image data.

  When “0” is set in the least significant bit, there is no parallax image data or combined image data. When “1” is set, parallax image data or combined image data exists.

  Next, the bit (bit1) next to the least significant bit indicates whether paint image data exists. When “0” is set in the bit 1, there is no paint image data. When “1” is set, paint image data exists.

  The remaining 6 bits of the 1 byte are reserved bits, and values are set as necessary when a data type is added. When not added, the initial value “0” is set in each bit (bit 2 to bit 7).

  Further, although the case where bit2 to bit7 according to the present embodiment are reserved bits has been described as an example, the present invention is not limited to this example. For example, bit2 indicates whether or not 2D characters are included in the paint image data. A value indicating the presence or absence of a 3D character is set in bit3, a value indicating the presence or absence of a 2D figure is set in bit4, and a value indicating the presence or absence of a 3D figure is set in bit5 , Bit6 may be set to a value indicating the presence or absence of a 2D frame, and bit7 may be set to a value indicating the presence or absence of a 3D frame.

  In addition, the paint image data is attached with hierarchical information indicating the depth of the image. It is also possible to manage the hierarchy information of the paint data as the left-right direction offset value of the paint data. The hierarchical information is information such as relatively managing the degree of overlap of paint image data when there are a plurality of paint image data.

  In the hierarchy information, the hierarchy becomes lower as the level increases, such as level 1, level 2, level 3,. The level 1 hierarchy is the top hierarchy. Therefore, the level 1 hierarchy information is higher than the level 2 hierarchy information.

  For example, if the paint image data composed of the characters “A” of the level 1 and the paint image data composed of the characters “B” of the level 2 overlap, the layer information of the paint image data “A”. Since “A” and “B” overlap each other, “A” paint image data is displayed.

  Also, as shown in FIG. 16, when the “paint input” button displayed on the GUI screen 220 is pressed, the paint image data is generated and displayed in the paint image 224, the paint image 225, or the main area 222. A painted image of a frame of a stereoscopic image (a frame hatched with a star mark such as “☆”) or the like can be displayed.

  In the main area 222, a paint image made up of an arbitrary character or symbol or the like is displayed so as to be superimposed on the stereoscopic image displayed in the main area 222. When the “paint input” button is pressed and an appropriate character or symbol or the like is designated, the paint image of the designated character or the like is superimposed on the stereoscopic image so as to be viewed stereoscopically.

  Therefore, as shown in FIG. 29, when the “paint input” button is pressed, the image conversion unit 139 generates paint image data and performs a fitting step of fitting the paint image data into the stereoscopic image data. (S612). The paint image displayed in the main area 222 can be moved to an arbitrary position by dragging and dropping the mouse. FIG. 29 is a flowchart showing an outline of the image processing method according to the present embodiment.

  Further, since the hierarchy of the paint image 224-1 is lower than the hierarchy of the paint image 224-2, the paint image 224-2 is displayed in the overlapping portion. The level set in the hierarchical information of the paint image 224-1 is raised by selecting the paint image 224-1 by clicking the mouse and pressing the “near” button displayed on the GUI screen 220.

  When the level of the hierarchy of the paint image 224-1 is increased, the paint image 224-1 is displayed at the overlapping portion with the paint image 224-2. Further, when the “near” button is pressed in a state where the paint image 224-1 is selected, the paint image 224-1 becomes an image in which the sense of perspective approaches. It should be noted that the “far” button 143 is pressed when the level of the hierarchy information is lowered and the perspective when the paint image 224 is stereoscopically viewed is reduced.

  Furthermore, as shown in FIG. 16, the hierarchical information associated with the arrangement of the objects in the stereoscopic image is obtained by associating the arrangement of the objects in the viewpoint image for the right eye and the viewpoint image for the left eye with the hierarchical information. Accordingly, the paint image 225 can be displayed as an index like a balloon.

  The paint image 225 is paint image data indicating an object in the stereoscopic image in a pop-up format. For example, the paint image 225-1 shown in FIG. 16 displays an index such as characters of the object “palm tree”, and the paint index image 225-2 includes an index such as characters of the object “yacht” in three dimensions. Appear to jump out.

  The paint image 225 is displayed when the object is selected by clicking an appropriate object in the stereoscopic image with the input unit. Note that the degree of three-dimensional pop-up of the paint image 225 is displayed based on the hierarchical information attached to the displayed paint image 225.

  Here, when the level of the hierarchy is not set in the hierarchy information attached to the paint image 225, an appropriate hierarchy can be selected by clicking the “near” button or the “far” button as described above. Level is set in the hierarchy information.

  In the setting of the hierarchy information according to the present embodiment, the level of the hierarchy may be set from a value calculated based on a color difference at corresponding points in the viewpoint images of the L image and the R image.

  In the frame of the paint image 225, characters such as “palm tree” are input, or a figure such as “palm tree” is pasted. Note that characters or graphics corresponding to the object are defined in advance in an object table (not shown) or the like, and can be automatically associated with the frame of the paint image 225 in the object table simply by clicking the object. It may be the case that a character or a figure is set. The object table is stored in a storage unit (not shown) provided in the imaging device 100 or the computer device 150.

  Although the case where characters such as “yacht” or “palm tree” are displayed in the paint image 225 according to the present embodiment has been described as an example, the present invention is not limited to this example. The paint image 225 can be implemented even when a graphic such as “yacht” or “palm tree” is displayed.

  Further, the paint image 225-1 and the paint image 225-2 differ in the degree to which the paint image 225 projects three-dimensionally according to the hierarchical information associated with the object arrangement.

  Since the object of the paint image 225-1 is “palm tree” and the object of the paint index image 225-2 is “yacht”, in the arrangement relationship of both objects in the stereoscopic image, the above “palm tree” The “tree” object is in front.

  In other words, the level of the hierarchical information of the paint image 225-1 corresponding to the “palm tree” object is lower than the level of the hierarchical information of the paint image 225-2 corresponding to the “yacht” object. The hierarchy is up.

  Therefore, the display is such that the paint image 225-1 on the “palm tree” pops out in front of the paint image 225-2 on the “yacht” (the degree of popping out is higher). Is done. Note that the present invention is not limited to this example, and the paint image 225 may be displayed in a planar manner. When displaying in a plane, the display of the paint image 225 may be enlarged as the hierarchy in the hierarchy information is higher.

  The paint image 225 is displayed according to the hierarchical information, and the recognition of the correspondence between the object and the paint image 225 is further deepened by displaying the paint image 225 according to the hierarchical information. , Visibility of an object configured in a stereoscopic image is increased.

  In addition, since the 3D image has a wider displayable area in three dimensions and the image is displayed three-dimensionally than the case where the paint image 225 is a planar 2D image, visibility is improved.

  In addition, the case where the stereoscopic pop-up degree of the paint image 225 is determined based on the hierarchical information associated with the object in the stereoscopic image according to the present embodiment has been described as an example, but is limited to this example. However, the present invention can be implemented even when the three-dimensional depth of the sound source is determined based on the hierarchical information, such as 3D sound.

  Therefore, when an object is selected by clicking or the like using the input unit, the three-dimensional position of the object is determined based on the hierarchical information attached to the paint image 225 corresponding to the object. appear. Note that the sound is output via a speaker provided in the stereoscopic display unit 140.

(Cut out viewpoint image data)
Next, extraction of viewpoint image data according to the present embodiment will be described. Cutting out viewpoint image data according to the present embodiment includes (1) cutting out by setting a cutting image area and (2) cutting out by setting an effective area.

(Cut image area)
(1) The size, position, offset value, and the like of the cut-out image area are stored in the “CropSizeX”, “CropSizeY”, “CropOffsetX”, and “CropOffsetY” tag fields of the image control information described above. .

  The tags “CropSizeX” and “CropSizeY” indicate the size (horizontal direction (X), vertical direction (Y)) of the clipped image area. The tags “CropOffsetX” and “CropOffsetY” indicate the offset (horizontal direction (X), vertical direction (Y)) of each clipped image area.

  As shown in FIG. 30, the byte lengths of the “CropSizeX” and “CropSizeY” tags are 4 bytes, respectively, and the byte lengths of the “CropOffsetX” and “CropOffsetY” tags are each 6 bytes.

  The area size that can be set in the fields “CropSizeX” and “CropSizeY” is 1 to 4294967295 (dot). In addition, the value set in the “CropOffsetX” and “CropOffsetY” tag fields is set to the X coordinate of the horizontal viewpoint position in the first byte of the least significant digit in the above field, and in the second byte. The Y coordinate of the viewpoint position in the vertical direction is set, and the offset value is set in the third to sixth bytes. Note that “CropOffsetY” includes the correction amount of the vertical shift of each viewpoint image. If the correction amount of shift is unknown, the offset value is “0”.

  By setting the cut-out image area, it is possible to designate a portion to be displayed as a stereoscopic image among the viewpoint image data of each viewpoint generated by the imaging of the imaging device 100. Designating a portion to be displayed as a stereoscopic image from the viewpoint image is referred to as image clipping. The specified cutout area is set as a cutout image area.

  Note that, with respect to the other combined image areas excluding the cutout image area according to the present embodiment, for example, when stored as it is or after specifying the cutout image area, the combined image area other than the cutout image area is used. May be deleted without memorizing.

  Here, the cutout image region according to the present embodiment will be described with reference to FIG. FIG. 31 is an explanatory diagram showing a schematic configuration of a cut-out image area according to the present embodiment. Note that the parallax image illustrated in FIG. 31 is described as an example of a combined image of two-viewpoint images (L image, R image) arranged in parallel, but is not limited to such an example.

  As shown in FIG. 31, the cut-out image area represented by the area hatched in gray is designated by the coordinates of the upper left position and the image size. The coordinates of the upper left position correspond to the mark “x” shown in FIG.

  Note that there are regions (margin regions) represented by broken lines on both the left and right sides of the right cut image region (R image) shown in FIG. When generating a stereoscopic image from the cut image areas of the L image and the R image, since the horizontal parallax is necessary, the cut image area is shifted in the horizontal direction.

  In the above case, there is no image in the area other than the cut-out image area after the cut-out, and therefore, by using the margin area, an appropriate stereoscopic image can be generated on both sides of the stereoscopic image. Normally, if there is no margin area, there will be no images on both sides, so the surrounding area will be black, etc.

  Since the clipped image area is specified for each viewpoint image data of each viewpoint, there are as many fields for setting the value of each offset (“CropOffset”) as the number of viewpoints, but the size of each clipped image area is the same. Therefore, one field for setting the size (“CropSize”) exists in common.

  When setting the offset value, in the “CropOffsetX” and “CropOffsetY” fields, the X coordinate of the horizontal viewpoint position of the first byte and the Y coordinate of the vertical viewpoint position of the second byte are set to “ By specifying (-1, -1) ", when there are a plurality of viewpoint images, the value set in the third byte is an offset value for all viewpoint images. By the above batch designation, the viewpoint position setting process can be made more efficient. For example, when the viewpoint position of the first “CropOffsetX” and “CropOffsetY” is set to “(−1, −1)”, it is possible to exemplify a case where batch designation is performed.

  Therefore, by specifying “(−1, −1)” for the horizontal viewpoint position and the vertical viewpoint position “(X, Y)”, it is possible to specify offsets collectively.

  Here, for example, among 16 viewpoint images of a plurality of viewpoints (4 × 4 viewpoints), only the horizontal offset value whose viewpoint position is “(2, 3)” is “8” and viewpoints from the remaining 15 viewpoints. A case where the horizontal offset value of the image is set to “10” will be described.

  In the above case, if the “CropOffsetX” field is set as (1st byte, 2nd byte, 3rd byte), as the offset of the viewpoint image of 15 viewpoints, the horizontal offset is (-1, 1, 10) can be set at one time, and the horizontal offset of the remaining viewpoint position “(2, 3)” may be “(2, 3, 8)”.

  In addition, when stereoscopic image data is generated from parallax image data (combined image data) based on each cut-out viewpoint image data by designating a cut-out image area, for example, a telephoto-captured object (object )) Or the like, a stereoscopic image with a strong stereoscopic effect (high pop-out degree) may be displayed on the stereoscopic display unit 140. For example, a warning screen (not shown) is displayed. Thus, by adjusting the offset, parallax can be averaged to balance the three-dimensional effect or weaken the three-dimensional effect, thereby reducing eye strain.

  Note that “CropOffsetX” (x1, x2) shown in FIG. 31 indicates the horizontal offset of the cutout region, “CropOffsetY” (y1, y2) indicates the vertical offset of the cutout region, and “CropSizeX”. “(L) indicates a horizontal length (size) for cutting out a predetermined area as a cut-out area of the combined image, and“ CropSizeY ”(H) indicates a predetermined area as a cut-out area of the combined image. Indicates the length (size) in the vertical direction for cutting out.

  Next, the cut-out process of the cut-out image area according to the present embodiment will be described.

  First, when a subject is imaged by the imaging apparatus 100, the synthesis unit 131 of the imaging unit 101 synthesizes viewpoint image data (L image data and R image data) transmitted from the imaging element 130 to generate combined image data. To do.

  Next, the image encoding unit 132 encodes the combined image data transmitted from the synthesizing unit 131, and the image control information generating unit 133 generates image control information together with the encoding process. After the encoding and generating process is completed, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  When the computer device 150 reads the designated combined image data and image control information from the recording medium 135, the data separation unit 136 separates the combined image data and the image control information into the image decoding unit 137 and the image separation unit. It transmits to each part 138.

  When the decoded combined image data and the image control information are transmitted to the image separation unit 138, as shown in FIG. 32, in order to set a cut-out image region in the stereoscopic display unit 140 of the computer apparatus 150. The editing screen 300 is displayed.

  In the main area 222 of the editing screen 300, a rectangle 302-1 indicating the cut-out image area is displayed together with the viewpoint image. The size of the rectangle 302-1 can be changed by dragging the mouse. Note that the size and position of the rectangle 302-1 is the cut-out image area.

  The rectangle 302-1 can be moved by dragging and dropping the mouse. The editing screen 300 according to the present embodiment will be described by taking an example in which viewpoint images are displayed one by one. However, the editing screen 300 is not limited to this example, and multiple viewpoint images are displayed simultaneously. Even if it is, it can be implemented. Further, when the rectangle 302-1 or the rectangle 302-2 is cut out, the depth offset amount is automatically adjusted within the range of the rectangle, and the offset amount can be optimized and stored in the image control information. , When the viewpoint image is read and displayed, the offset amount can be referred to from the image control information. In addition, for example, an image is cut out according to the offset amount stored in the image control information, or a stereoscopic image of an area where a remark point is set is enlarged / reduced according to the stored offset amount. The offset, cutout, and remark points can be used in cooperation.

  When the “Crop determination” button shown in FIG. 32 is clicked by clicking with a mouse or the like, the image separation unit 138 selects “CropSizeX”, “CropSizeY”, “CropOffsetX”, “CropOffsetX”, And a value to be set in the tag field of “CropOffsetY” is obtained, and a value is set in each tag.

  After the setting, the image separation unit 138 further cuts out each viewpoint image data (L image, R image) in the combined image data according to the obtained value (cutout image region). The cut out L image data and R image data are transmitted to the image conversion unit 139 and converted into stereoscopic image data.

  By setting the value of the clipped image area in the field of each tag, the clipped image area is determined again from the L image data and the R image data, and then converted into stereoscopic image data. By reading the set value, it is possible to quickly and efficiently determine the cut-out image area.

  Note that when the image conversion unit 139 converts the L image data and the R image data into the stereoscopic image data, the cut-out L image data and R image data are converted into a screen size (size) of the stereoscopic display unit 140. Each is enlarged (resized) so as to conform to, and converted into stereoscopic image data.

  Further, when the L image data and the R image data are enlarged by the image conversion unit 139, for example, when the parallax between the L image data and the R image data exceeds a predetermined threshold, or the average value of the parallax Etc., a warning screen (not shown) described above is displayed.

  Further, the image conversion unit 139 performs offset adjustment so that the parallax is reduced within the threshold range. For example, the parallax between the L image data and the R image data according to the present embodiment is exemplified by vertical parallax or horizontal parallax.

  Furthermore, a modified example of the cutout process of the cutout image area according to the present embodiment will be described.

  When a subject having a depth, for example, located far away from a stereoscopic image is cut out and displayed on the stereoscopic display unit 140, the displayed stereoscopic image includes many stereoscopic subjects. It is very difficult to see and tends to cause eye strain.

  As shown in FIG. 32, a rectangle 302-2 is set in the stereoscopic image displayed on the editing screen 300 by an input unit such as a mouse. A rectangle 302-2 has a yacht with depth and a sky in the background as a cut-out image area.

  Next, when the “Clip Determination” button shown in FIG. 32 is clicked by clicking with the mouse or the like, “CropSizeX”, “CropSizeY”, “CropOffsetX”, and “CropOffsetY” are based on the size and position of the rectangle 302-2. The value to be set in the tag field of “” is obtained, and the value is set in each tag.

  After the setting, the image separation unit 138 further cuts out each viewpoint image data (L image, R image) in the combined image data according to the obtained value (cutout image region). The cut out L image data and R image data are transmitted to the image conversion unit 139.

  The image conversion unit 139 converts the minimum amount of parallax into, for example, “0” by matching the transmitted L image data and R image data with corresponding points, and the maximum amount of parallax is 65 mm which is the interocular distance. Adjust the parallax so that it does not exceed. Note that if the parallax exceeds the interocular distance of 65 mm, the 3D display fails and a stereoscopic image cannot be displayed. When matching corresponding points, a confirmation screen for matching corresponding points is displayed on the stereoscopic display unit 140 in advance.

  After the adjustment, the image conversion unit 139 combines the L image data and the R image data with the stereoscopic image data and displays it on the stereoscopic display unit 140. Since the displayed stereoscopic image has at least a minimum parallax of “0”, it includes an image area with a stereoscopic effect and an image area with a reduced stereoscopic effect. A stereoscopic image with high image quality. Therefore, editing processing such as fine adjustment by rotating the stereoscopic image is quickly performed. Conventionally, it is often difficult for a user to visually recognize a stereoscopic image after being cut out, which hinders editing processing and the like.

  In the corresponding point matching, first, corresponding points included in the R image data and L image data are extracted. The corresponding points may be extracted either when the user is extracted or automatically extracted. The corresponding points to be extracted are, for example, three places, but are not limited to this example.

  Corresponding points are coincident points corresponding to the same subject in the L image and the R image. That is, the corresponding point needs to exist in all of the L image and the R image. The corresponding points are not limited to the case of two viewpoints, and may be a plurality of viewpoint images.

  As a specific example of the corresponding points, for example, in the case of two viewpoints, when the subject is a cup, the cup photographed in the L image and the cup photographed in the R image represent the photographing position, angle, size, and the entire image. Of these, a point where the range occupied by the cup coincides with the same degree is taken as the corresponding point.

  When extracting corresponding points, it is possible to efficiently extract appropriate corresponding points by increasing the contrast of each viewpoint image or binarizing each viewpoint image. For binarization of images, for example, a threshold value of 1 or 2 or more is set for the luminance (brightness) of each pixel of the viewpoint image. If the threshold value is less than the threshold value, “0” (black) is set. "(White)".

  When the corresponding points are extracted, the corresponding corresponding points are connected for each viewpoint image, and the distance between the corresponding points is obtained. Furthermore, the amount of parallax is calculated | required by taking the difference of each distance between the calculated | required corresponding points. For example, in the case of two viewpoints, three corresponding points are extracted from the L image and the R image, the corresponding points are connected in the L image and the R image, and the distances (A, B, C between the three corresponding points are connected. ) Is required. Next, each parallax amount is obtained by taking the absolute value of the difference (AB, AC, BC) based on the distance between the three corresponding points thus obtained. In addition, although extraction of a corresponding point is performed about the far periphery part with a depth among viewpoint images, it is not limited to this example.

  Next, the minimum parallax amount is selected from the obtained parallax amounts, and it is confirmed whether or not the minimum parallax amount is 0 mm. When the minimum amount of parallax is not 0 mm, the display size of each viewpoint image is adjusted so that the minimum amount of parallax is 0 mm. Note that a warning that there are many stereoscopic images is displayed on the stereoscopic display unit 140 before the parallax amount is adjusted.

  The stereoscopic image may be displayed as it is after the minimum parallax amount is adjusted to 0 mm. However, the present invention is not limited to this example, and the maximum parallax amount is selected from the obtained parallax amounts. It may be confirmed whether or not the amount exceeds the interocular distance of 65 mm. When the maximum parallax amount does not exceed 65 mm, the maximum parallax amount of the cut-out image area is determined.

  Further, when the maximum parallax amount exceeds 65 mm, the display size of each viewpoint image, the image left-right shift amount, and the like are adjusted so that the maximum parallax amount is smaller than 65 mm, and the maximum parallax amount is determined. The display size is also determined. Before the adjustment, a warning that the maximum parallax amount exceeds 65 mm and a stereoscopic image cannot be displayed is displayed on the stereoscopic display unit 140.

  When the parallax of the clipped image area is adjusted, a stereoscopic image of the clipped image area is displayed on the stereoscopic display unit 140.

  As in the modification example of the cutout process of the cutout image area according to the present embodiment, even when a remark point (reference attention point) is set and enlarged display is performed, corresponding point matching or parallax adjustment is performed. Later, even if a stereoscopic image is displayed, it can be implemented. A warning or the like is also displayed on the screen.

(Effective area)
Next, (2) the size, position, offset value, etc. of the effective area are set in the tag fields of “ValidSizeX”, “ValidSizeY”, “ValidOffsetX”, and “ValidOffsetY” in the image control information described above. Is done.

  “ValidSizeX” and “ValidSizeY” indicate the size of the effective area (horizontal direction (X), vertical direction (Y)). “ValidOffsetX” and “ValidOffsetY” indicate offsets (horizontal direction (X), vertical direction (Y)) with respect to each effective area.

  As shown in FIG. 33, “ValidSizeX” and “ValidSizeY” are each 6 bytes, and “ValidOffsetX” and “ValidOffsetY” are each 6 bytes.

  In addition, the value set in the fields of the tags “ValidSizeX” and “ValidSizeY” is set such that the X coordinate of the horizontal viewpoint position is set in the first byte of the least significant digit in the above field, and the second byte contains , The Y coordinate of the vertical viewpoint position is set, and an offset value is set in the third to sixth bytes.

  In addition, the value set in the fields of the tags “ValidOffsetX” and “ValidOffsetY” is set such that the X coordinate of the horizontal viewpoint position is set in the first byte of the least significant digit and the second byte in the field. , The Y coordinate of the vertical viewpoint position is set, and the offset value to the viewpoint position is set in the third to sixth bytes.

  For example, each viewpoint image captured in 3D, such as imaging by the imaging device 100 including the optical adapter 105, may not be suitable as a stereoscopic image due to the influence of the optical adapter 105 or the like. The region of the end portion varies depending on the type of the optical adapter 105. Therefore, by setting the effective area, it is possible to specify a range of an effective area as a stereoscopic image.

  Here, the effective region according to the present embodiment will be described with reference to FIG. FIG. 34 is an explanatory diagram showing a schematic configuration of an effective area according to the present embodiment. Note that the combined image shown in FIG. 34 is described as an example of a parallel arrangement of two viewpoint images (L image, R image), but is not limited to such an example, and the combined image is in an intersecting arrangement. It may be the case.

  As shown in FIG. 34, the size, position, offset value, and the like of the effective area represented by gray hatching are specified based on the coordinates of the upper left position and the image size. The coordinates of the upper left position correspond to the mark “x” shown in FIG. The reference position is not limited to the upper left position, and may be another appropriate position.

  The value of each offset needs to be specified for each viewpoint image, and the effective area size value may be different for each viewpoint image, and is set for each viewpoint image. Note that the processing of the protruding portion when the cut-out image region protrudes from the effective region depends on the image processing device such as the computer device 150.

  Note that “x1” shown in FIG. 34 is set in the “ValidOffsetX” tag field of the L image, “x2” is set in the “ValidOffsetX” tag field of the R image, and “y1” is “L1” of the “L” image. “ValidOffsetY” is set in the tag field, “y2” is set in the “ValidOffsetY” tag field of the R image, “L1” is set in the “ValidSizeX” tag field of the L image, and “L2” is The tag field of “ValidSizeX” of the R image is set, “H1” is set to the tag field of “ValidSizeY” of the L image, and “H2” is set to the tag field of “ValidSizeY” of the R image. .

  “ValidOffsetX” (x1, x2) shown in FIG. 31 indicates a horizontal offset from the upper left vertex in the effective area of the R image and L image, and “ValidOffsetY” (y1, y2) indicates the R image and L image. An offset in the vertical direction from the upper left vertex in the effective area is indicated. “ValidSizeX” (L1, L2) indicates a horizontal length (size) of the effective area of the L image or the R image, and “ValidSizeY” (H1, H2). ) Indicates the length (size) of the effective area of the L image or R image in the vertical direction.

  Next, the effective area extraction processing according to the present embodiment will be described.

  First, the type of adapter model number to be attached to the imaging apparatus 100 is selected in advance. When the adapter model number is selected, it is set from the GUI screen 221 displayed on the display unit (not shown) of the imaging apparatus 100. Note that the selection of the adapter model number type is almost the same as the description in the adapter model number information according to the present embodiment, and thus detailed description thereof is omitted.

  As shown in FIG. 27, on the GUI screen 221, each adapter model name consisting of “No. 1” to “No. 4”, an adapter selection unit 223 for selecting an adapter model number, and a “select” button are displayed. Is done.

  Next, the adapter model name of the optical adapter 105 to be mounted on the imaging apparatus 100 is selected from the displayed adapter model names using the jog dial or the like as the input unit, and when the “select” button is pressed, the adapter selection unit 223 is selected. Generates adapter model number information in which the adapter model name is set. Note that the adapter model number information is stored in a storage unit (not shown) of the imaging apparatus 100 or the like.

  Next, when the subject is imaged using the imaging device 100, the synthesis unit 131 of the imaging unit 101 synthesizes the viewpoint image data (L image data and R image data) transmitted from the imaging element 130, and combines the combined images. Generate data.

  Further, the image control information generation unit 133 generates image control information for the generated combined image data, and uses the adapter model number information stored in the storage unit of the imaging apparatus 100 as the “user data area tag” of the image control information. Set in the tag field.

  Further, the image control information generation unit 133 generates effective area information from which end portions that are not suitable as stereoscopic images are removed based on the adapter model number information of the optical adapter 105 stored in the storage unit.

  Next, the image control information generation unit 133 sets the effective area information in the tag fields of “ValidSizeX”, “ValidSizeY”, “ValidOffsetX”, and “ValidOffsetY” of the image control information.

  The image control information generation unit 133 generates the effective area information based on the adapter model number information. However, the image control information generation unit 133 is not limited to the example, and the effective area information is preliminarily input by the input unit via the effective area setting GUI screen. Even if it is stipulated, it can be implemented.

  Next, the image encoding unit 132 encodes the combined image data transmitted from the synthesizing unit 131, and the image control information generating unit 133 generates image control information together with the encoding process. After the encoding and generating process is completed, the combined image data and the image control information are multiplexed by the data multiplexing unit 134 and recorded on the recording medium 135.

  The computer device 150 reads the combined image data and the image control information from the recording medium 135.

  Next, the data separation unit 136 of the computer device 150 separates the combined image data into image control information.

  Next, the image separation unit 138 receives the combined image data decoded by the image decoding unit 137 and the image control information.

  As shown in FIG. 34, the image separation unit 138 is based on the values (effective area information) set in the tag fields of “ValidSizeX”, “ValidSizeY”, “ValidOffsetX”, and “ValidOffsetY” of the image control information. An area corresponding to the effective area is cut out from the combined image area.

  If a value (effective area information) is not set in the fields of “ValidSizeX”, “ValidSizeY”, “ValidOffsetX”, and “ValidOffsetY” of the image control information, the image separation unit 138 uses the actual image area. The entire certain viewpoint image data is transmitted to the image conversion unit 139.

  The image conversion unit 139 converts each viewpoint image data (L image data, R image data) transmitted from the image separation unit 138 into stereoscopic image data. Therefore, a stereoscopic image is displayed on the stereoscopic display unit 140.

  Note that the cut out effective area is determined by the adapter model number information of the optical adapter 105 attached to the imaging apparatus 100. Therefore, the image separation unit 138 can appropriately adjust the offset between the L image data and the R image data in accordance with the effective area determined based on the adapter model number information.

(Reference attention point)
In addition to designating a cutout area or an effective area, in the viewpoint image data, gazing point (reference attention point, remark point) information related to the gazing point that is the center when displaying a stereoscopic image is displayed in the image described above. It can be set to a “remark point tag” tag included in the control information.

  The center of the stereoscopic display unit 140 is not necessarily limited to the case where the center of the stereoscopic image is arranged and displayed at the center of the stereoscopic display unit 140. (Reference attention point) can be set. In particular, when the size of the stereoscopic display unit 140 such as a portable terminal or a PDA is smaller than the size of the stereoscopic image data, a stereoscopic image with high visibility is set by setting a portion to be watched by the user as a reference attention point. Can be displayed.

  The reference attention point is represented by “(x, y)” in which the horizontal direction is the x-axis and the vertical direction is the y-axis, and can be set in the effective area or the cut-out image area shown in FIGS. is there. The tag field of “remark point tag” consists of 6 bytes, the 1st to 3rd bytes of the least significant digit indicate the x coordinate of the reference attention point, and the 4th to 6th bytes are the reference attention point. Indicates the y coordinate.

  Next, reference attention point setting processing according to the present embodiment will be described.

  First, when a subject is imaged by the imaging apparatus 100, the synthesis unit 131 of the imaging unit 101 combines viewpoint image data (L image data and R image data) transmitted from the imaging element 130 to generate combined image data. To do.

  Next, when the image encoding unit 132 encodes the combined image data and the image control information generation unit 133 generates image control information corresponding to the combined image data, the combined image data and the image control information are converted into a data multiplexing unit. Multiplexed by 134 and recorded on the recording medium 135.

  The computer device 150 reads the designated combined image data and image control information from the recording medium 135. Next, the data separation unit 136 separates the combined image data and the image control information, and transmits them to the image decoding unit 137 and the image separation unit 138, respectively.

  When the decoded combined image data and the image control information are transmitted to the image separation unit 138, as shown in FIG. 35, a reference attention point is set in the stereoscopic display unit 140 of the computer apparatus 150. An edit screen 320 is displayed.

  In the main area 222 of the editing screen 320, a mark 303 indicating a reference attention point when converting each viewpoint image into a stereoscopic image is displayed. The mark 303 can be moved by dragging and dropping the mouse. Note that the reference attention point according to the present embodiment will be described using an example in which each viewpoint image is the same, but the reference attention point is not limited to this example, and the reference attention point is different between the viewpoint images of each viewpoint. Even if it is a case, it can be implemented. If they are different, for example, it is necessary to perform selection processing for selecting any one of a plurality of attention reference points.

  When the “remark point determination” button for setting the reference attention point shown in FIG. 35A is pressed by clicking the mouse or the like, the image separation unit 138 sets the “remark point tag” of the image control information for the combined image data. The value of the reference attention point “(x, y)” of the mark 303 is set in the tag field.

  After the setting, the image separation unit 138 separates the combined image data into L image data and R image data. The separated L image data and R image data are transmitted to the image conversion unit 139 and converted into stereoscopic image data. Note that when separating, for example, when a cut image area is set, the image separation unit 138 also performs L image data and R image data cut processing, and the like.

  When converting to the stereoscopic image data, the image conversion unit 139 follows the reference attention point set in the “remark point tag” so that the reference attention point is located at the center of the stereoscopic display unit 140. , L image data and R image data are moved. In addition, the image conversion unit 139 enlarges the L image data and the R image data and converts them into stereoscopic image data so as to adapt to the screen size of the stereoscopic display unit 140. Therefore, as shown in FIG. 35B, an enlarged stereoscopic image in which the reference attention point is located at the center of the stereoscopic display unit 140 is displayed.

  Further, when the stereoscopic image data is enlarged and displayed, the image conversion unit 139 offsets the reference attention point set in the “remark point tag” and the partial region of the viewpoint image centered on the reference attention point. Based on the above, a stereoscopic image of a partial region can be enlarged and displayed. Note that the offset position can adjust the front-rear relationship of the depth of the stereoscopic image. For example, in the case of two viewpoints, the offset position is adjusted by adjusting the vertical position and the horizontal shift amount (corresponding to the pop-out amount) when the R image and the L image are superimposed and displayed stereoscopically. For example, referring to FIG. 32, the rectangle 302-2 which is a partial region is a stereoscopic image of the ship, but the image of the ship is behind the stereoscopic image of the person of the rectangle 302-1. positioned. Therefore, since the image of the ship is in the back, when the mark 303 is set as the reference attention point in the “ship” displayed in the rectangle 302-2 and enlarged display is performed, the image conversion unit 139 The offset position is adjusted so that the image of the ship of the rectangle 302-2 is displayed in the front, and the stereoscopic image of the ship of the rectangle 302-2 is enlarged and displayed. Thus, when the “ship” image displayed in the back of the “person” shown in FIG. 32 is enlarged, the stereoscopic image of the person displayed in the rectangle 302-1 by one enlargement process. Because it is displayed three-dimensionally in the front, it is easy to see. On the contrary, when the mark 303 is set to a person displayed in the area of the rectangle 302-1, the offset position adjustment amount is small and enlarged because it is originally located and displayed in the front.

  When the reference attention point is set in the tag field of “remark point tag”, the stereoscopic image shown in FIG. 35B is further enlarged around the reference attention point. Even if the center point to be enlarged again is not defined or the reference attention point is not set, the stereoscopic image can be rapidly enlarged around the attention point.

  When the stereoscopic image is enlarged around the reference attention point, the editing screen 320 is switched to a warning screen (not shown), and a warning message such as “There is a possibility that stereoscopic viewing may be difficult” is displayed. Is displayed.

  The warning screen is displayed because the parallax increases due to the enlargement of the stereoscopic image, so that it is difficult to visually recognize the stereoscopic image. Moreover, it is because eye strain is promoted by expansion of parallax.

  The reference attention point according to the present embodiment has been described by taking as an example a case where the reference attention point is a center point when a stereoscopic image is enlarged. However, the reference attention point is not limited to such a case. For example, it may be the center point when the visual image is reduced.

  Further, as described above, the remark point (reference remark point) is set as the reference remark point according to the present embodiment, as in the modified example of the cut-out processing of the cut-out image area according to the present embodiment. Even in the case of enlarging display or the like, the present invention can be implemented even when displaying a stereoscopic image after matching corresponding points or adjusting parallax. A warning or the like is also displayed on the screen.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, this invention is not limited to this example. It is obvious for a person skilled in the art that various changes or modifications can be envisaged within the scope of the technical idea described in the claims, and these are naturally within the technical scope of the present invention. It is understood that it belongs.

  In the above embodiment, the case where the two-viewpoint image is combined with the stereoscopic image has been described as an example. However, the present invention is not limited to this example. For example, the present invention can be implemented even in the case of a plurality of viewpoint images having three or more viewpoints.

  The present invention can be applied to a stereoscopic image processing apparatus that can generate a stereoscopic image.

FIG. 1 is an explanatory diagram showing a schematic configuration of an imaging apparatus equipped with the optical adapter according to the present embodiment. FIG. 2 is an explanatory diagram showing a configuration example of the optical adapter of FIG. FIG. 3 is an explanatory diagram illustrating a parallax image captured by an imaging apparatus equipped with the optical adapter according to the present embodiment. FIG. 4 is an explanatory diagram showing a schematic configuration of stereoscopic image display by the projector according to the present embodiment. FIG. 5 is an explanatory diagram showing a schematic configuration of the computer apparatus according to the present embodiment. FIG. 6 is an explanatory diagram for explaining processing for generating a stereoscopic image by combining the L image and the R image according to the present embodiment. FIG. 7 is an explanatory diagram showing a stereoscopic image according to the present embodiment. FIG. 8 is a diagram for explaining an overview of stereoscopically viewing a stereoscopic image according to the present embodiment. FIG. 9 is a block diagram showing a schematic configuration of the stereoscopic image processing apparatus according to the present embodiment. FIG. 10 is a modification of the stereoscopic image processing apparatus shown in FIG. FIG. 11 is an explanatory diagram showing an outline of the data structure of the stereoscopic image file according to the present embodiment. FIG. 12 is an explanatory diagram showing a schematic configuration of combined image data according to the present embodiment. FIG. 13 is an explanatory diagram illustrating a schematic configuration of a viewpoint image when captured from a plurality of viewpoints according to the present embodiment. FIG. 14 is an explanatory diagram showing a schematic configuration of the viewpoint image arrangement in the present embodiment. FIG. 15 is a flowchart showing an outline of the viewpoint image processing method according to the present embodiment. FIG. 16 is an explanatory diagram showing a schematic configuration example of the display screen in the present embodiment. FIG. 17 is an explanatory diagram showing a schematic configuration of uniform rotation arrangement of viewpoint images in the present embodiment. FIG. 18 is an explanatory diagram showing a schematic configuration of the individual rotational arrangement of viewpoint images in the present embodiment. FIG. 19 is an explanatory diagram showing a schematic configuration of uniform rotation of the viewpoint image and mirror inversion arrangement in the present embodiment. FIG. 20 is an explanatory diagram showing a schematic configuration of viewpoint image individual rotation and mirror inversion arrangement in the present embodiment. FIG. 21 is a flowchart showing an outline of the combined image recording process according to the present embodiment. FIG. 22 is an explanatory diagram showing a schematic configuration of the environment information setting screen according to the present embodiment. FIG. 23 is a flowchart showing an outline of a stereoscopic image display process according to the present embodiment. FIG. 24 is a flowchart showing an outline of a stereoscopic image display process according to the present embodiment. FIG. 25 is an explanatory diagram showing an outline of values set in the “Picture Rotate NM” tag field according to the present embodiment. FIG. 26 is a flowchart illustrating an outline of a modified example of the image display process of the stereoscopic image according to the present embodiment. FIG. 27 is an explanatory diagram showing a schematic configuration of a display screen example according to the present embodiment. FIG. 28 is an explanatory diagram showing a schematic configuration of the optical adapter according to the present embodiment. FIG. 29 is a flowchart showing an outline of the image processing method according to the present embodiment. FIG. 30 is an explanatory diagram showing an outline of values set in the “CropSize” or “CropOffset” tag field according to the present embodiment. FIG. 31 is an explanatory diagram showing a schematic configuration of a cut-out image area according to the present embodiment. FIG. 32 is an explanatory diagram showing a schematic configuration of a display screen example according to the present embodiment. FIG. 33 is an explanatory diagram showing an outline of values set in the field of the “ValidSize” or “ValidOffset” tag according to the present embodiment. FIG. 34 is an explanatory diagram showing a schematic configuration of an effective area according to the present embodiment. FIG. 35 is an explanatory diagram showing a schematic configuration of a display screen example according to the present embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101: Imaging part 131: Synthesis | combination part 132: Image encoding part 133: Image control information generation part 134: Data multiplexing part 135: Recording medium 136: Data separation part 137: Image decoding part 138: Image separation part 139: Image conversion part 140: Stereoscopic display unit

Claims (5)

An optical adapter for acquiring a plurality of viewpoint images having parallax with each other as a parallax image in an intersection arrangement by one shooting, and an acquisition unit for acquiring the plurality of viewpoint images by one shooting;
A record for recording information including environment information for designating which side of the right viewpoint image is arranged as information indicating a combined image of a plurality of viewpoint images acquired by the acquisition unit and characteristics of the optical adapter. And
Among the combined images recorded by the recording unit, an effective region is recorded by the recording unit excluding a portion unsuitable for generating a stereoscopic image from each region of the combined image corresponding to each viewpoint image. A cutout unit that cuts out based on information indicating the characteristics of the optical adapter,
A separation unit that separates the combined image into the plurality of viewpoint images with respect to an effective region cut out by the cutout unit;
An image generation unit that generates a stereoscopic image in an arrangement specified by the environment information from a plurality of viewpoint images separated by the separation unit;
A stereoscopic image processing apparatus comprising: The acquisition unit acquires a right viewpoint image and a left viewpoint image,
The cutout unit cuts out a region of the combined image excluding regions corresponding to an upper side portion, a lower side portion, and a left side portion of the right viewpoint image that has deteriorated due to the optical adapter as the effective region. The stereoscopic image processing apparatus described. For each of the right viewpoint image and the left viewpoint image separated by the separation unit, a trapezoidal distortion or a trapezoidal distortion or a chromatic aberration is corrected based on the information indicating the characteristics of the optical adapter recorded by the recording unit. A correction unit,
The stereoscopic image processing apparatus according to claim 2, wherein the image generation unit generates a stereoscopic image from the right viewpoint image and the left viewpoint image corrected by the correction unit. Wherein the correction unit, when correcting the trapezoidal distortion and the chromatic aberration, when correcting the chromatic aberration, black out a location where color is blurred, the stereoscopic image processing apparatus according to claim 3.   The stereoscopic image processing apparatus according to claim 1, wherein the optical adapter is at least one of a two-mirror adapter, a four-mirror adapter, a mountain prism adapter, or an anamorphic lens adapter or any combination thereof.

Images