JP2020088843A - Imaging system, terminal, and program - Google Patents

Abstract

To project a desired display object regardless of the attitude of an imaging unit.SOLUTION: An imaging system comprises: an imaging unit that picks up an image; a detection unit that detects the attitude of the imaging unit; a correction unit that, based on the attitude of the imaging unit and a display portion designated on the image, corrects the image picked up by the imaging unit to project the display portion; and a display unit that displays the image after the correction.SELECTED DRAWING: Figure 6

Description

The present invention relates to an imaging system, a terminal and a program.

A camera that captures an image of a wide-angle field of view such as a hemisphere and a celestial sphere is known.

For example, there is a spherical image capturing system including a spherical camera (for example, Patent Document 1). This spherical camera has two image forming optical systems each having a fisheye lens, and two image pickup elements. The omnidirectional camera synthesizes the images picked up by the two image pickup devices to generate an image having a solid angle of 4π steradians, that is, an omnidirectional image. The omnidirectional image is an image showing all the directions that can be viewed from the imaging point. The omnidirectional video image capturing system of Patent Document 1 corrects the inclination of the omnidirectional image to correct the vertical direction even when the omnidirectional camera is imaged in a tilted state.

The spherical moving image capturing system of Patent Document 1 generates a spherical image in which the tilt of the spherical camera is corrected. However, when the orientation of the omnidirectional camera with respect to the subject changes, the subject projected in the center of the omnidirectional image may move to another position. For example, when the omnidirectional image generated by the omnidirectional video image capturing system is simultaneously delivered to a plurality of viewers through communication, or when an object not desired by the viewer is projected in the center of the omnidirectional image. There is. That is, the orientation of the omnidirectional image may be the orientation that the viewer does not want.

Then, the imaging system, the terminal, and the program of the present disclosure aim to project a desired display target regardless of the orientation of the imaging unit such as a camera.

An imaging system according to an embodiment of the present invention is based on an imaging unit that captures an image, a detection unit that detects the orientation of the imaging unit, an orientation of the imaging unit, and a display portion designated in the image. A correction unit that corrects the image captured by the image capturing unit so that the display portion is displayed, and a display unit that displays the corrected image.

According to the technique of the present disclosure, it is possible to capture a desired display target regardless of the posture of the image capturing unit.

FIG. 2 is a diagram showing an example of the configuration of the imaging system according to the first embodiment. Side view showing an example of the imaging device according to the first embodiment FIG. 3 is a diagram showing an example of a hardware configuration of the image pickup apparatus according to the first embodiment. The figure which shows an example of the hardware constitutions of the terminal which concerns on Embodiment 1. FIG. 3 is a block diagram showing an example of a functional configuration of the image pickup apparatus according to Embodiment 1. Block diagram showing an example of the functional configuration of the terminal according to the first embodiment FIG. 3 is a diagram showing an example in which a flat image frame is converted into a spherical image frame by the display image generation unit according to the first embodiment. The figure which shows an example of the image on the optical axis center in the spherical image frame of FIG. 7A. The figure which shows an example of the designated area which concerns on Embodiment 1. The figure which shows an example of the center point of the designated area|region which concerns on Embodiment 1. FIG. 3 is a diagram showing an example of a display image of a terminal before and after a posture change of the imaging device according to the first embodiment Sequence diagram showing an example of an image distribution operation in the imaging system according to the first embodiment Flowchart showing an example of the operation of the imaging apparatus according to the first embodiment The flowchart which shows an example of operation|movement of the terminal which concerns on Embodiment 1. The figure which shows an example of the designated point and designated direction which concern on Embodiment 2. FIG. 6 is a diagram showing an example of display images of a terminal before and after a posture change of the imaging device according to the second embodiment

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, and duplicate description will be omitted.

(Embodiment 1)
<Structure of Imaging System 1000>
The configuration of the imaging system 1000 according to the first embodiment will be described. FIG. 1 is a diagram showing an example of the configuration of an imaging system 1000 according to the first embodiment. As shown in FIG. 1, the imaging system 1000 includes an imaging device 10 and a terminal 20. In the present embodiment, imaging system 1000 includes one imaging device 10 and one terminal 20, but may include two or more imaging devices 10 and may include two or more terminals 20.

The imaging device 10 and the terminal 20 communicate with each other. The captured image captured by the imaging device 10 is sent to the terminal 20 as image data, and the viewer can view the sent image data on the terminal 20. The imaging device 10 and the terminal 20 may be indirectly connected via the communication network 30, or may be directly connected via wired communication or wireless communication. In the present embodiment, the imaging device 10 and the terminal 20 are connected to each other via the communication network 30. For example, when two or more terminals 20 are provided, the imaging device 10 can simultaneously deliver the captured image to the two or more terminals 20 as image data. The communication network 30 may be the Internet, a wired LAN (Local Area Network), a wireless LAN, a mobile communication network, a telephone line communication network, or any other communication network using wired or wireless communication. In the present embodiment, communication network 30 is the Internet.

The imaging device 10 and the terminal 20 may each be configured by one or more devices. When the device is composed of two or more devices, the two or more devices may be arranged in one device or may be separately arranged in two or more separate devices. In the present description and claims, "device" can mean not only one device but also a system composed of a plurality of devices.

The imaging device 10 includes an imaging unit 11A (see FIG. 2) including a camera that captures a still image and/or a moving image that is a digital image, and transmits the captured image to the terminal 20. The moving image may be a continuous moving image composed of a plurality of frames of images captured continuously, or a time-lapse moving image composed of a plurality of still images captured at intervals. .. In the present embodiment, the imaging device 10 captures continuous moving images and transmits the image data of a plurality of frames forming the moving images to the terminal 20.

Hereinafter, one frame image is also referred to as “image frame”. Further, in the present specification and claims, “image” means not only the image itself, but also image data that is data indicating an image and an image signal that is a signal indicating an image. The “image frame” may mean not only the one-frame image itself, but also data indicating one-frame image and a signal indicating one-frame image.

The imaging unit 11A may be configured by a camera that captures an image with a general viewing angle, or may be configured by a camera that captures a wide-angle image with a viewing angle of up to about 180°, and the imaging angle of more than 180°. It may be configured by a camera that captures an image of a super wide angle that is a viewing angle. In the present embodiment, the image capturing unit 11A is described as a spherical camera having a spherical field of view, but the present invention is not limited to this.

The terminal 20 is a computer device having a communication function and capable of displaying an image. Examples of the terminal 20 include a notebook PC (Personal Computer), a smart device such as a mobile phone, a smartphone and a tablet terminal, a game machine, a PDA (Personal Digital Assistant), a wearable PC, a desktop PC, a video conference terminal, and an IWB (Interactive White). Board: a white board having an electronic blackboard function that enables mutual communication). In the present embodiment, the terminal 20 is described as a notebook PC, but the present invention is not limited to this.

<Structure of Imaging Device 10>
The configuration of the imaging device 10 will be described. FIG. 2 is a side view showing an example of the imaging device 10 according to the first embodiment. As shown in FIG. 2, the imaging device 10 includes an imaging unit 11A and a housing 11B. The imaging unit 11A includes cameras 11a and 11b and a controller 11c. The housing 11B houses the cameras 11a and 11b and the controller 11c.

Each of the cameras 11a and 11b is a super wide-angle camera having a field of view exceeding a hemisphere (angle of view exceeding 180°). The camera 11a includes an image forming optical system 11aa and a first image sensor 11ab, and the camera 11b includes an image forming optical system 11ba and a second image sensor 11bb.

The imaging optical system 11aa and the imaging optical system 11ba respectively form an incident image on the light receiving surfaces of the first image sensor 11ab and the second image sensor 11bb. The imaging optical system 11aa and the imaging optical system 11ba each include optical elements such as a lens, a prism, a filter, and an aperture stop. The lens is a lens having an angle of view of more than 180°, such as a fisheye lens.

The image forming optical system 11aa and the image forming optical system 11ba are arranged so that the light incident directions on the lenses are opposite to each other. The optical axis centers of the lenses of the imaging optical system 11aa and the imaging optical system 11ba configure the same optical axis center LA. The image forming optical system 11aa and the image forming optical system 11ba have the same specifications, and they are combined in opposite directions so that their optical axis centers coincide with each other.

The first image sensor 11ab and the second image sensor 11bb include a plurality of light receiving elements arranged in an array. The light receiving element outputs a pixel value indicating the intensity of the received light. An example of a pixel value is a brightness value. The light receiving element constitutes a plurality of pixels for picking up an image in each of the first image pickup element 11ab and the second image pickup element 11bb. The first image pickup device 11ab and the second image pickup device 11bb sequentially use, as image frames, image signals indicating the light distribution of the images formed on the light receiving surfaces by the image forming optical system 11aa and the image forming optical system 11ba, respectively, as the controller 11c. Output to.

Examples of the first image sensor 11ab and the second image sensor 11bb are a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In the present embodiment, the first image sensor 11ab and the second image sensor 11bb are the same image sensor having the same resolution and the same image sensor size, but the present invention is not limited to this.

The controller 11c is a computer device including a microcomputer or the like. The controller 11c synthesizes the two image frames imaged by the first image sensor 11ab and the second image sensor 11bb into one image frame, and an image with a solid angle of 4π steradian (hereinafter, also referred to as “all sphere image”). ) Generate a frame. The omnidirectional image is an image showing all directions centered on the imaging point. For example, the celestial sphere image frame may show a plane image formed by projecting a spherical surface onto a cylinder by an equirectangular projection and developing the cylinder. Then, a continuous celestial sphere moving image is composed of consecutive celestial sphere image frames.

Here, the direction of the optical axis center LA of the imaging optical system 11aa and the imaging optical system 11ba is defined as the X-axis direction. The direction that is the longitudinal direction of the elongated rectangular parallelepiped housing 11B and is perpendicular to the X axis is defined as the Z axis direction. The direction perpendicular to the X axis and the Z axis is defined as the Y axis direction. The three-dimensional coordinate system including the XYZ axes constitutes the camera coordinate system of the image pickup apparatus 10. The camera coordinate system is fixed to the image pickup apparatus 10 and changes the orientation of the coordinate axes according to the change in the attitude of the image pickup apparatus 10.

<Hardware Configuration of Imaging Device 10>
The hardware configuration of the imaging device 10 will be described. FIG. 3 is a diagram illustrating an example of a hardware configuration of the image pickup apparatus 10 according to the first embodiment. As illustrated in FIG. 3, the imaging device 10 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a moving image compression block 104, and a first imaging element 11ab. A second image sensor 11bb, an image processing block 106, a memory 107, an external storage interface (I/F: Interface) 108, a motion sensor 109, a communication I/F 110, and a wireless NIC (Network Interface Card). 111 and 111 as constituent elements. Note that not all of these constituent elements are essential. Further, each of the above components is connected to each other via, for example, a bus. The above components may be connected via either wired communication or wireless communication.

The CPU 101 is composed of a processor and the like, and controls the operation and overall operation of each unit of the imaging device 10. For example, the CPU 101 constitutes the controller 11c shown in FIG. The ROM 102 is configured by a non-volatile semiconductor storage device or the like, and stores various programs and various parameters that operate in the imaging device 10. A RAM (Random Access Memory) 103 is composed of a volatile semiconductor memory device or the like and is used as a work area of the CPU 101. The RAM 103 provides a storage area for temporarily storing data when performing various kinds of signal processing and image processing. The memory 107 stores various information such as data used in various programs and images captured by the first image sensor 11ab and the second image sensor 11bb. The memory 107 is configured with a storage device such as a volatile or non-volatile semiconductor memory, an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory 107 may include the ROM 102 and/or the RAM 103.

The program is stored in advance in the ROM 102, the memory 107, or the like. The program is read by the CPU 101 from the ROM 102 or the memory 107 or the like into the RAM 103 and expanded. The CPU 101 executes each coded instruction in the program expanded in the RAM 103. The program is not limited to the ROM 102 and the memory 107, and may be stored in a recording medium such as a recording disk. In addition, the program may be transmitted via a wired network, a wireless network, broadcasting, or the like, and may be loaded into the RAM 103.

The functions realized by the CPU 101 described above may be realized by a program execution unit such as the CPU 101, a circuit, or a combination of the program execution unit and the circuit. For example, such a function may be realized by an LSI (Large Scale Integration) that is an integrated circuit. Such a function may be individually implemented on a single chip, or may be implemented on a single chip so as to include a part or all of the functions. As an LSI, an FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, a reconfigurable processor in which connection and/or settings of circuit cells inside the LSI can be reconfigured, or a plurality of devices for specific purposes An ASIC (Application Specific Integrated Circuit) in which functional circuits are integrated may be used.

The image processing block 106 is connected to the two first image pickup elements 11ab and the second image pickup element 11bb, and receives image signals of images picked up by the first image pickup element 11ab and the second image pickup element 11bb, respectively. The image processing block 106 is configured to include an ISP (Image Signal Processor) and the like, and performs shading correction, Bayer interpolation, white balance correction, and the like on the image signals input from the first image sensor 11ab and the second image sensor 11bb. Performs gamma correction and other processing. The image processing block 106 also synthesizes the two image frames acquired from the first image sensor 11ab and the second image sensor 11bb to generate a spherical image frame. That is, the image processing block 106 combines two image frames imaged by two image pickup elements at the same time or at a similar time into one spherical image frame. More specifically, in the image processing block 106, first, each spherical image including each complementary hemispherical portion is generated from each captured image configured as a planar image. Then, the two spherical images including the respective hemispherical portions are aligned based on the matching of the overlapping regions and the images are combined to generate a spherical image frame including the entire spherical image.

The moving picture compression block 104 is an MPEG-4 AVC/H. H. It is a codec block that compresses and decompresses moving images such as H.265. The moving image compression block 104 compresses the data of the spherical image frame generated by the image processing block 106 into moving image format data.

The motion sensor 109 includes an acceleration sensor, an angular velocity sensor (also called “gyro sensor”), and a geomagnetic sensor. The acceleration sensor detects accelerations in the three XYZ axis directions. The angular velocity sensor detects the angular velocities around the three axes. The geomagnetic sensor detects the magnetic fields in the three axis directions. Such a motion sensor 109 constitutes a 9-axis attitude sensor. The motion sensor 109 calculates the roll angle around the X axis, the pitch angle around the Y axis, and the yaw angle around the Z axis in the imaging device 10 using the detection values of the sensors. The roll angle indicates the posture of the image pickup device 10 in the rolling direction, the pitch angle indicates the posture of the image pickup device 10 in the pitching direction, and the yaw angle indicates the posture of the image pickup device 10 in the yawing direction.

The motion sensor 109 calculates a roll angle, a pitch angle, and a yaw angle each time an image frame is captured. The motion sensor 109 may calculate the roll angle, the pitch angle, and the yaw angle between the image frames, or may calculate the roll angle, the pitch angle, and the yaw angle with respect to the reference posture of the imaging device 10. The motion sensor 109 may output the detection value of each sensor to the CPU 101, and the CPU 101 may calculate the roll angle, the pitch angle, and the yaw angle. The motion sensor 109 may include at least an acceleration sensor and an angular velocity sensor.

An external storage such as a storage medium is connected to the external storage I/F 108. For example, the external storage I/F 108 controls reading and writing with respect to an external storage such as a memory card which is an example of a storage medium inserted in the memory card slot.

The communication I/F 110 is an interface that is connected to another device or device and transmits/receives information. For example, the communication I/F 110 may be connected to another device via wired communication or wireless communication. The communication I/F 110 may include a USB (Universal Serial Bus) connector or the like.

The wireless NIC 111 is a device for communicating with other devices such as the terminal 20 via the communication network 30. The wireless NIC 111 may include a connection terminal, a communication circuit, and the like.

<Hardware configuration of terminal 20>
The hardware configuration of the terminal 20 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the terminal 20 according to the first embodiment. As shown in FIG. 4, the terminal 20 includes a CPU 201, a ROM 202, a RAM 203, an input device 204, a display 205, a wireless NIC 206, a communication I/F 207, a memory 208, and an external storage I/F 209. Include as a component. Note that not all of these constituent elements are essential. Further, although the above-described constituent elements are connected to each other via, for example, a bus, they may be connected to each other via wired communication or wireless communication.

The configurations of the CPU 201, the ROM 202, and the RAM 203 are similar to those of the CPU 101, the ROM 102, and the RAM 103. The CPU 201 controls the operation of each unit of the terminal 20 and the overall operation. The ROM 202 stores various programs operating on the terminal 20. The RAM 203 is used as a work area for the CPU 201.

The memory 208 stores various information such as data used in various programs and images acquired from the imaging device 10. The configuration of the memory 208 is similar to that of the memory 107.

The function implemented by the CPU 201 may be implemented by a program execution unit such as the CPU 201, a circuit, or a combination of the program execution unit and the circuit.

The wireless NIC 206 is a device for communicating with other devices such as the imaging device 10 via the communication network 30. The wireless NIC 206 may include a connection terminal, a communication circuit, and the like.

The communication I/F 207 is an interface that is connected to another device or device and that transmits and receives information. For example, the communication I/F 207 may be connected to another device via wired communication or wireless communication. The communication I/F 207 may include a USB connector or the like.

An external storage such as a storage medium is connected to the external storage I/F 209. The external storage I/F 209 controls reading and writing with respect to the connected storage medium.

The input device 204 is a device that receives an operation input by a user. The input device 204 may include input devices such as buttons, dials, keys, a mouse, a touch panel, and a microphone for voice input.

The display 205 displays various screens under the control of the CPU 201. The display 205 may be a liquid crystal panel, an organic EL (Electroluminescence) display, an inorganic EL display, or the like. The display 205 may include a speaker for audio output. The display 205 may be a touch panel that also serves as the input device 204.

<Functional configuration of the imaging device 10>
The functional configuration of the imaging device 10 will be described. FIG. 5 is a block diagram showing an example of a functional configuration of the image pickup apparatus 10 according to the first embodiment. As shown in FIG. 5, the imaging device 10 includes an imaging control unit 11, a moving image compression unit 12, a posture detection unit 13, a posture data generation unit 14, and a communication unit 15 as functional components.

The function of the imaging control unit 11 is realized by the CPU 101, the image processing block 106, and the like. The imaging control unit 11 controls the first imaging element 11ab and the second imaging element 11bb to capture an image, and acquires the data of the image frame captured by the first imaging element 11ab and the second imaging element 11bb. Furthermore, the imaging control unit 11 combines the two image frames imaged by the first imaging element 11ab and the second imaging element 11bb at the same time or at a similar time to generate a celestial sphere image frame. The imaging control unit 11 sequentially outputs the generated spherical image frames to the moving image compression unit 12.

The function of the moving picture compression unit 12 is realized by the CPU 101, the moving picture compression block 104, and the like. The moving image compression unit 12 compresses the omnidirectional image frame acquired from the imaging control unit 11 into moving image format data and outputs the data to the communication unit 15. Here, the moving image compression unit 12 is an example of a generation unit.

The function of the posture detection unit 13 is realized by the CPU 101 and the like. The posture detection unit 13 controls the motion sensor 109 and acquires the posture information of the imaging device 10 from the motion sensor 109. That is, the posture detection unit 13 detects the posture of the imaging device 10. The posture information may be time-series data of each detection value of the motion sensor 109. The time-series data is data that includes a detection value and its detection time in association with each other. For example, the posture detection unit 13 may obtain the detection value detected by the motion sensor 109 at the timing when the omnidirectional image frame is captured by the first image sensor 11ab and the second image sensor 11bb from the time-series data. Good. The above-mentioned timing may be the time at which the spherical image frame is captured or a time in the vicinity thereof.

The function of the posture data generation unit 14 is realized by the CPU 101 and the like. The posture data generation unit 14 uses the posture information acquired from the posture detection unit 13 to generate a parameter indicating the posture as posture data indicating the posture of the imaging device 10. Examples of parameters indicating the attitude are yaw angle, pitch angle, and roll angle. The attitude data generation unit 14 outputs the attitude data to the communication unit 15.

The function of the communication unit 15 is realized by the CPU 101, the wireless NIC 111, and the like. The communication unit 15 associates the moving image format data with the attitude data of the imaging device 10, and transmits the data to the terminal 20 via the communication network 30. At this time, the communication unit 15 may associate the frame of each time included in the moving image format data with the parameter of each time included in the posture data based on the time. Here, the communication unit 15 is an example of a transmission unit.

<Functional configuration of terminal 20>
The functional configuration of the terminal 20 will be described. FIG. 6 is a block diagram showing an example of a functional configuration of terminal 20 according to the first embodiment. As shown in FIG. 6, the terminal 20 includes a communication unit 21, a moving image expansion unit 22, a display image generation unit 23, a posture data acquisition unit 24, a display viewpoint correction unit 25, and a display control unit 26. Included as a functional component.

The function of the communication unit 21 is realized by the CPU 201, the wireless NIC 206, and the like. The communication unit 21 receives, that is, acquires the moving image format data transmitted by the communication unit 15 of the imaging device 10 and the attitude data of the imaging device 10. The communication unit 21 outputs the data in the moving image format to the moving image developing unit 22 and outputs the posture data to the posture data acquisition unit 24. Here, the communication unit 21 is an example of an acquisition unit.

The function of the moving image developing unit 22 is realized by the CPU 201 and the like. The moving image decompression unit 22 decompresses the compressed moving image format data into displayable image frame data and outputs the data to the display image generation unit 23. That is, the moving image expanding unit 22 expands the moving image format data into individual image frame data. Here, the moving image developing unit 22 is an example of a developing unit.

The function of the posture data acquisition unit 24 is realized by the CPU 201 and the like. The attitude data acquisition unit 24 acquires the attitude data of the imaging device 10 via the communication unit 21 and outputs it to the display viewpoint correction unit 25.

The function of the display image generation unit 23 is realized by the CPU 201 and the like. The display image generation unit 23 converts the image frame data developed from the moving image data into the data in which the image frame is pasted on the surface of the virtual sphere. That is, the display image generation unit 23 attaches the spherical image frame represented by the equirectangular projection to the spherical surface and converts it into a three-dimensional model. The display image generation unit 23 outputs the data of the omnidirectional image frame attached to the spherical surface to the display viewpoint correction unit 25.

FIG. 7A is a diagram showing an example of conversion from a spherical image frame to an image frame attached to a spherical surface of a three-dimensional model by the display image generating unit 23 according to the first embodiment. 7B is a diagram showing an example of an image on the optical axis center LA in the spherical spherical image frame of FIG. 7A. An example of this image is obtained by performing perspective projection processing on the three-dimensional model attached to the spherical surface.

As shown in FIG. 7A, the planar spherical image frame developed from the moving image format data is pasted, that is, projected onto the spherical surface of a virtual solid sphere CS centered on the imaging device 10. Converted to a spherical image frame. At this time, for example, the display image generation unit 23 includes a three-dimensional coordinate system (camera coordinate system) including the XYZ axes set in the imaging device 10 and a three-dimensional coordinate system including the XsYsZs axes set in the solid sphere CS. May be converted so that and correspond to each other, that is, the axes match. The three-dimensional coordinate system including the XsYsZs axes is the world coordinate system. The world coordinate system is a fixed coordinate system irrespective of the posture of the imaging device 10. The Zs axis direction is the gravity direction, that is, the vertical direction, and the Xs axis direction and the Ys axis direction are the horizontal directions.

In such a spherical image frame, an image of a region T ₀ centered on the intersection of the optical axis center LA of the image pickup device 10 and the spherical surface is represented as an image shown in FIG. 7B. For all axes passing through the center of the solid sphere CS, an image of a region centered on the intersection of the axis and the spherical surface is similarly represented.

The function of the display viewpoint correction unit 25 is realized by the CPU 201 and the like. The display viewpoint correction unit 25 corrects the direction of the optical axis center in the spherical image frame based on the posture data. Further, the display viewpoint correction unit 25 forms a celestial sphere image frame which is a planar image frame obtained by developing a spherical image frame around the corrected optical axis center by an equidistant cylindrical projection method, and the display control is performed. It is output to the unit 26. Here, the display viewpoint correction unit 25 is an example of a correction unit.

The function of the display control unit 26 is realized by the CPU 201 and the like. The display control unit 26 causes the display 205 to display the image frame acquired from the display viewpoint correction unit 25. For example, the display control unit 26 sequentially displays a plurality of image frames forming a moving image and displays the moving image. Here, the display control unit 26 is an example of a display unit.

Details of the display viewpoint correction unit 25 will be described. For example, on the screen of the omnidirectional image captured by the imaging device 10 displayed on the display 205 of the terminal 20, the user uses the input device 204 to specify a display portion such as a point or a region. At this time, the display viewpoint correction unit 25 corrects the spherical image frame so that the spherical image that maintains the position of the display portion on the screen is displayed. Even when the imaging device 10 is rotated and the direction of the optical axis center LA is changed, the display viewpoint correction unit 25 maintains the position of the display portion with respect to the spherical image displayed on the screen. Therefore, the display viewpoint correction unit 25 maintains the viewpoint position to the display portion. The method of designating the display portion is not limited to points and areas, and any method may be used.

A case where a rectangular designation area T is designated as a display portion on the screen of the display 205 will be described. In this case, the display viewpoint correction unit 25 projects the designated area T on the spherical surface of the solid sphere CS as shown in FIG. 7A. Further, the display viewpoint correction unit 25 calculates the position of the center point CP of the projected designated area T on the spherical surface of the solid sphere CS.

FIG. 8 is a diagram showing an example of the designated area according to the first embodiment. FIG. 9 is a diagram showing an example of the center point of the designated area according to the first embodiment. FIG. 9 shows a part of a spherical image frame on the solid sphere CS.

As shown in FIG. 8, a virtual camera IC that envisions the designated area T and is located at the center of the solid sphere CS is assumed. The intersection of the optical axis IA of the virtual camera IC and the spherical surface of the solid sphere CS is the center point CP of the designated area T. Specifically, as shown in FIG. 9, the viewing angle of the virtual camera IC with respect to the diagonal line angle of view 2L of the designated area T is α. The center point CP is the intersection of the optical axis IA, which is a bisector of the viewing angle α, and the diagonal line of the designated area T corresponding to the diagonal line angle of view 2L.

For example, when the posture of the image pickup apparatus 10 changes by the roll angle θr, the pitch angle θp, and the yaw angle θy due to the movement of the image pickup apparatus 10, the direction of the optical axis center LA of the image pickup apparatus 10 also changes. The optical axis center LA after the change is referred to as an optical axis center LA1. The direction of the optical axis IA and the position of the center point CP also change similarly to the optical axis center LA. The changed optical axis IA is referred to as an optical axis IA1. The center point CP moves to the intersection point CP1 between the optical axis IA1 and the solid sphere CS.

The display viewpoint correction unit 25 rotates the spherical image frame on the solid sphere CS so that the position of the center point CP on the solid sphere CS does not change before and after the posture change. Specifically, the display viewpoint correction unit 25 rotates the image frame of the spherical surface after the posture change about the center of the solid sphere CS so as to move the center point CP1 to the center point CP. At this time, the display viewpoint correction unit 25 may match the optical axis IA1 with the optical axis IA, or may match the optical axis center LA1 with the optical axis center LA.

For example, the display viewpoint correction unit 25 corrects each parameter of the spherical image frame after the posture change by performing coordinate conversion to move the optical axis IA1 to the optical axis IA. In the spherical image frame after coordinate conversion, the position of the center point CP1 on the solid sphere CS is the same as the center point CP before the posture change.

In this way, the display viewpoint correction unit 25 performs coordinate conversion based on the roll angle θr, the pitch angle θp, and the yaw angle θy indicating the attitude change of the image pickup apparatus 10 on the spherical image frame after the attitude change, and thereby the stereoscopic image is obtained. The position of the designated area T on the sphere CS is maintained, and the line of sight to the designated area T is maintained. Therefore, the display viewpoint correction unit 25 projects the image portion of the spherical surface after the posture change on the display portion based on the posture information of the image pickup apparatus 10, that is, the posture data and the information on the designated area T which is the display portion. To correct. Then, the display viewpoint correction unit 25 maintains the position of the display portion regardless of the posture change of the imaging device 10.

For example, FIG. 10 is a diagram showing an example of display images of the terminal 20 before and after the posture change of the imaging device 10 according to the first embodiment. As shown in FIG. 10, in the image 20a displayed by the terminal 20, the region T is designated by the user. After that, for example, when the imaging device 10 is rotated around the Z axis, the terminal 20 displays the image 20a in which the position of the center point CP of the region T is maintained at the same position during and after the rotation. That is, the terminal 20 continues to display the image 20a of the same viewpoint. The point CP may be designated by the user. Also in this case, the terminal 20 displays the image 20a in which the position of the point CP is maintained at the same position during and after the rotation. In addition, the image of the same viewpoint means that the regions T designated by the user before and after the posture change may include at least 75% or more of mutual overlap on the screen.

<Image distribution operation of imaging system 1000>
In the imaging system 1000, an operation in which the imaging device 10 distributes image data to the terminal 20 will be described. FIG. 11 is a sequence diagram showing an example of an image data distribution operation in the imaging system 1000 according to the first embodiment. The case where the terminal 20 receives the designation of the region T as a display portion from the user in the image displayed on the display 205 will be described below.

As shown in FIG. 11, the imaging device 10 acquires an image of each frame by capturing a moving image (step S1). At this time, the imaging device 10 combines the image frames acquired from the first imaging element 11ab and the second imaging element 11bb to generate a celestial sphere image frame.

Further, the imaging device 10 acquires the attitude data of the imaging device 10 via the motion sensor 109 (step S2). For example, the imaging device 10 detects the attitude data, which is time-series data, at the timing of imaging each spherical image frame (more specifically, the timing of the image frame acquired by the two imaging elements). The attitude data is acquired and the attitude data is associated with the spherical image frame. That is, the imaging device 10 associates the spherical image frame acquired at the same timing with the attitude data. The information associating the spherical image frame with the posture data may be included in the spherical image frame, may be included in the posture data, or may be included in both. Note that the timing includes the same time as a certain time and a time approximate to the time.

Next, the imaging device 10 performs encoding for compressing the omnidirectional image frame into moving image format data (step S3). Next, the imaging device 10 associates the moving image format data with the posture data, and transmits, that is, distributes to the terminal 20 (step S4).

Next, the terminal 20 receives the moving image format data and the attitude data, and performs decoding to expand the moving image format data into displayable image frame data (step S5). Further, the terminal 20 converts the image frame data expanded from the moving image data into an image frame to be attached to the spherical surface of the three-dimensional model.

Then, the terminal 20 acquires information on a viewpoint (also referred to as a viewpoint position or a designated area) which is a display portion designated by the user on the image displayed on the screen of the display 205 (step S6). The viewpoint information includes the position and size of the point or area designated by the user on the display image, and is shown as data including the coordinate value of the world coordinate system. May be In this example, the area T is designated. The designation may be performed by point-and-click or dragging with a mouse, which is an example of the input device 204, on the screen.

Further, the terminal 20 sequentially corrects the display area of the spherical image frame based on the posture data and the information of the designated area T (step S7). Specifically, the terminal 20 uses the light of the spherical image frame so that the position of the designated area T displayed on the display screen of the terminal 20 does not change regardless of the change in the attitude of the imaging device 10 indicated by the attitude data. Correct the orientation of the axis center.

Next, the terminal 20 performs projective transformation such as perspective projection transformation on the corrected spherical surface image frame to sequentially convert it into a planar spherical image frame, and the display 205 Is displayed (step 8).

<Operation of Imaging Device 10>
The operation of the imaging device 10 will be described. FIG. 12 is a flowchart showing an example of the operation of the image pickup apparatus 10 according to the first embodiment. Hereinafter, the processing of the imaging device 10 for one frame of image data will be described.

As shown in FIG. 12, the first image pickup device 11ab and the second image pickup device 11bb of the image pickup apparatus 10 start image pickup, and the image pickup control unit 11 is picked up by the first image pickup device 11ab and the second image pickup device 11bb, respectively. An image frame that is image data for one frame is acquired, that is, captured (step S101).

Next, the imaging control unit 11 performs image processing on the acquired image data (step S102). Specifically, the imaging control unit 11 combines the two image frames imaged by the first imaging element 11ab and the second imaging element 11bb to generate a spherical image frame.

Next, the posture detection unit 13 detects the posture of the imaging device 10 (step S103). Specifically, the posture detection unit 13 acquires, from the detection value of the motion sensor 109, the detection value detected at the timing when the spherical image frame is captured. Further, the posture data generation unit 14 uses the detection value acquired by the posture detection unit 13 to generate a parameter indicating the posture of the imaging device 10 as posture data.

Next, the moving picture compression unit 12 compresses one frame of image data into moving picture image data (step S104). Specifically, the moving picture compression unit 12 compresses the spherical image frame into moving picture format data. Here, the image data is compressed into a moving image format every frame, but it is compressed into a moving image format every predetermined number of frames, or all frames are compressed into a moving image format after shooting is finished. Any form may be used.

Next, the communication unit 15 transmits the moving image format image data to the terminal 20 (step S105).

Similarly, the communication unit 15 transmits the attitude data of the imaging device 10 to the terminal 20 (step S106). At this time, the communication unit 15 associates the moving image format image data with the posture data detected at the timing of capturing the moving image format image data. The communication unit 15 may include information associating the image data in the moving image format with the attitude data in the image data in the moving image format, in the attitude data, or in both. In addition, orientation data may be added and associated as metadata of moving image format image data. Further, when the posture data is added as the metadata, instead of transmitting the posture data to the terminal 20 as in step S106, when the posture data is compressed into moving image format data in step S104, it corresponds to each image frame. The posture data may be added as metadata and compressed into moving image format data.

Next, when the image pickup by the first image pickup device 11ab and the second image pickup device 11bb is finished and the process for all the image frames is finished (Yes in step S107), the image pickup control unit 11 finishes the series of processes. If there is an unprocessed image frame (No in step S107), the process returns to step S101.

<Operation of terminal 20>
The operation of the terminal 20 will be described. FIG. 13 is a flowchart showing an example of operation of terminal 20 according to the first exemplary embodiment. In the following, when the terminal 20 is displaying the image distributed from the imaging device 10 on the display 205, the operation of the terminal 20 after the viewpoint position (specified region) is specified by the user via the input device 204 will be described. explain.

As shown in FIG. 13, the terminal 20 receives the designation of the viewpoint position from the user on the displayed image (step S201). For example, the viewpoint position is information indicating a point or region designated on the image via the input to the input device 204, including coordinate values in the world coordinate system.

Next, the communication unit 21 receives one frame of image data, which is moving image format data, from the imaging device 10 (step S202).

Similarly, the communication unit 21 receives the attitude data of the imaging device 10 from the imaging device 10 (step S203). Note that, as described above, when the posture data is received as the metadata of the moving image format data, the step S203 is omitted. Then, the posture data acquisition unit 24 acquires the received posture data.

Next, the moving image expanding unit 22 expands one frame of image data in the moving image format in the RAM 103 (step S204). Specifically, the moving image expanding unit 22 expands the image data in the moving image format into the data of the displayable image frame.

Next, the display image generation unit 23 converts the displayable image frame data into an image frame projected on the spherical surface of the three-dimensional model. That is, the display image generation unit 23 generates an image frame for display (step S205). Here, for example, when outputting the omnidirectional image frame to a plane image, perspective projection conversion is performed.

Next, the display viewpoint correction unit 25 corrects the display viewpoint in the spherical image frame based on the posture data (step S206). At this time, the display viewpoint correction unit 25 performs correction by associating the posture data and the spherical image frame, which have the same detection timing of the posture data and the imaging timing of the image forming the spherical image frame. Then, the display viewpoint correction unit 25 changes the spherical image frame so that the viewpoint position specified in step S201 does not change on the screen of the terminal 20 due to the change in the attitude of the imaging device 10 indicated by the attitude data. The direction, that is, the direction of the optical axis center LA is corrected. Therefore, the display viewpoint correction unit 25 corrects the viewpoint position, which may change due to the change in the attitude of the imaging device 10, to the position before the change in the attitude. Furthermore, the display viewpoint correction unit 25 forms a spherical image frame on the spherical surface of the three-dimensional model with the corrected optical axis center as the center.

Next, the display control unit 26 causes the display 205 to display the corrected image frame acquired from the display viewpoint correction unit 25. That is, the display control unit 26 displays the area T at the corrected viewpoint (step S207). Therefore, the image of the same viewpoint position is displayed on the display 205 even if the posture of the imaging device 10 changes.

Next, when the display control unit 26 finishes displaying all the image frames transmitted from the imaging device 10 on the display 205 (Yes in step S208), the series of processes ends, and if not finished (step S208). No), the process returns to step S201.

In step S201, when the terminal 20 receives the change in the viewpoint position from the user, the terminal 20 uses the changed viewpoint position and proceeds to the processing in step S202 and subsequent steps. When the terminal 20 has not received the change of the viewpoint position from the user, the terminal 20 proceeds to the processing of step S202 and subsequent steps using the viewpoint position currently set. Upon receiving the cancellation of the viewpoint position from the user, the terminal 20 proceeds to step S208.

<Effects>
In the imaging system 1000 according to the first embodiment as described above, the imaging unit 11A that captures an image, the attitude detection unit 13 that detects the attitude of the imaging unit 11A, the attitude of the imaging unit 11A, and the image are designated. A display viewpoint correction unit 25 as a correction unit that corrects the image captured by the imaging unit 11A based on the display portion so as to project the display portion, and a display control unit that serves as the display unit that displays the corrected image. And 26.

According to the above configuration, the imaging system 1000 displays an image showing the display portion even if the attitude of the imaging unit 11A changes. For example, even when the orientation of the imaging unit 11A changes and the imaging direction changes, the imaging system 1000 projects the display portion in the displayed image. Therefore, the imaging system 1000 can project a desired display target regardless of the posture of the imaging unit 11A.

Further, in the image pickup system 1000 according to the first embodiment, the display viewpoint correction unit 25 is imaged by the image pickup unit 11A so as to maintain the position of the display portion with respect to the image regardless of the change in the posture of the image pickup unit 11A. The image may be corrected. According to the above configuration, the imaging system 1000 can display the display portion at the same position in the displayed image. With this, the user does not need to search for the display portion in the image, and can visually recognize the display portion without changing the viewpoint. Therefore, it becomes easy to visually recognize the display portion.

Further, in the imaging system 1000 according to the first embodiment, the attitude of the imaging unit 11A may include the attitudes of the imaging unit 11A in the yawing, pitching, and rolling directions. With the above configuration, the accuracy of detecting the posture of the image capturing unit 11A is improved. Therefore, the imaging system 1000 can accurately correct the image for displaying the display portion.

Moreover, the imaging system 1000 according to the first embodiment may include the imaging device 10 and the terminal 20. Then, the imaging device 10 includes the imaging unit 11A, the moving image compression unit 12 as a generation unit that generates moving image data from the plurality of images captured by the imaging unit 11A, the posture detection unit 13, the moving image data and the imaging unit 11A. The communication unit 15 may be included as a transmission unit that sends the posture of the above to the terminal 20. Further, the terminal 20 may include a moving image developing unit 22 as a developing unit that develops moving image data into individual images, a display viewpoint correcting unit 25, and a display control unit 26. According to the above configuration, the imaging system 1000 can deliver a moving image from the imaging device 10 to the terminal 20, and display a display portion in the delivered moving image.

Further, the terminal 20 according to the first embodiment specifies the image captured by the imaging unit 11A and the communication unit 21 as an acquisition unit that acquires information on the orientation of the imaging unit 11A, the orientation of the imaging unit 11A, and the image. The display viewpoint correction unit 25 that corrects the image acquired by the communication unit 21 so as to project the display portion based on the display portion that is displayed, and the display control unit 26 that displays the corrected image. According to the above configuration, the terminal 20 can achieve the same effect as that of the imaging system 1000.

(Embodiment 2)
In the imaging system according to the second embodiment, the processing for holding the viewpoint position for the image displayed on the display 205 is different from that of the first embodiment. Hereinafter, the second embodiment will be described focusing on the points different from the first embodiment, and the description of the same points as the first embodiment will be appropriately omitted.

The hardware configurations and functional configurations of the imaging device and the terminal according to the second embodiment are the same as those in the first embodiment, and thus the description thereof will be omitted. Furthermore, in the second embodiment, the reference numerals of the respective constituent elements are the same as those in the first embodiment.

In the present embodiment, on the image displayed on display 205 of terminal 20, the position of the viewpoint, which is the display portion, and the orientation of the image are specified by the user via input device 204. The designation method may be any designation method as long as the position of the viewpoint and the orientation of the image can be designated. For example, a point indicating the position of the viewpoint and a line indicating the orientation of the image may be designated. Alternatively, the area may be designated using a polygonal frame. The position of the viewpoint and the orientation of the image can be identified from the center of the polygon and one side.

A case where a point indicating the position of the viewpoint and a line indicating the orientation of the image are designated will be described. FIG. 14 is a diagram showing an example of designated points and designated directions according to the second embodiment. Similar to FIG. 9, FIG. 14 shows a part of a spherical image frame on the solid sphere CS. As shown in FIG. 14, the point CPa is designated as the designated point, and the line Da extending from the point CPa is designated as the line indicating the designated direction. The point on the opposite side of the point CPa on the line Da is the point CPb. This specified content is also specified in a form that includes coordinate values in the world coordinate system, but it may also be coordinate values on the spherical image.

Then, when the posture of the imaging device 10 changes, the point CPa and the line Da move together. The display viewpoint correction unit 25 of the terminal 20 rotates the spherical image frame on the solid sphere CS so that the position of the point CPa and the direction of the line Da on the solid sphere CS do not change before and after the posture change. , Correct each parameter of the spherical image frame.

For example, when the attitude of the imaging device 10 changes by the roll angle θr, the pitch angle θp, and the yaw angle θy, the optical axis IAa of the virtual camera IC passing through the point CPa changes to the optical axis IAa1 and the point CPa changes. Move to point CPa1 on IAa1. Further, the direction vector Va indicating the direction of the line Da changes to the direction vector Va1. The direction vector Va may be a vector formed by the points CPa and CPb.

Therefore, the display viewpoint correction unit 25 moves the spherical image frame after the posture change to the center of the solid sphere CS so that the point CPa1 is moved to the point CPa and the direction vector Va1 matches the direction vector Va. Rotate around. That is, the display viewpoint correction unit 25 performs coordinate conversion on the spherical image frame after the posture change to move the point CPa1 to the point CPa and match the direction of the direction vector Va1 with the direction of the direction vector Va. In the spherical image frame after the coordinate conversion, the position of the point CPa1 on the solid sphere CS is the same as the point CPa, and the direction vector Va1 is parallel to the direction vector Va.

For example, FIG. 15 is a diagram showing an example of display images of the terminal 20 before and after the posture change of the imaging device 10 according to the second embodiment. As shown in FIG. 15, in the image 20a displayed by the terminal 20, the point CPa and the line Da are designated by the user. Then, for example, when the imaging device 10 is rotated around the Y axis, the terminal 20 keeps the position of the point CPa in the same position and the direction of the line Da in the same direction during and after the rotation. The displayed image 20a is displayed. That is, the terminal 20 continues to display the image 20a with the same viewpoint and inclination.

Further, other configurations and operations of the imaging system according to the second embodiment are similar to those of the first embodiment, and thus the description thereof will be omitted. Then, according to the imaging system according to the second embodiment as described above, the same effect as that of the first embodiment can be obtained.

Further, in the imaging system according to the second embodiment, the display viewpoint correction unit 25 of the terminal 20 is imaged by the imaging unit 11A based on the attitude of the imaging unit 11A and the display portion and the display direction specified in the image. The image may be corrected to project the display portion in the display direction. According to the above configuration, the terminal 20 can display the image showing the display portion in the designated display direction even if the attitude of the imaging unit 11A changes. Therefore, the terminal 20 can project a desired display target in a desired orientation regardless of the posture of the imaging unit 11A.

In the imaging system according to the second embodiment, the display viewpoint correction unit 25 of the terminal 20 causes the image captured by the image capturing unit 11A so that the vertical axis of the image captured by the image capturing unit 11A is the zenith direction. May be corrected. The vertical axis of the image captured by the image capturing unit 11A may correspond to the vertical axis set in the image capturing unit 11A.

For example, the vertical axis set in the imaging unit 11A may be the Z axis. At this time, the display viewpoint correction unit 25 may use the vertical axis of the image corresponding to the Z axis, instead of the line Da designated by the user. Then, the display viewpoint correction unit 25 uses the point CPa designated by the user and the vertical axis of the image to maintain the position of the point CPa at the same position and maintains the direction of the vertical axis of the image in the zenith direction. As described above, the image captured by the image capturing unit 11A may be corrected. The zenith direction is the Zs axis direction in world coordinates. The tilt direction and tilt angle of the Z axis of the imaging unit 11A with respect to the Zs axis can be detected from the attitude of the imaging unit 11A. Therefore, the display viewpoint correction unit 25 can perform the correction similarly to the processing described above in the second embodiment.

With the above configuration, the terminal 20 can display an image showing the designated display portion in the zenith direction which is the actual vertical direction.

Further, the display viewpoint correction unit 25 may be capable of selecting a zenith correction function for correcting the image so that the vertical axis of the image is the zenith direction. For example, when the user executes the zenith correction function, that is, an ON command is input via the input device 204 of the terminal 20, the display viewpoint correction unit 25 may perform the zenith correction. Furthermore, when the user inputs a command to stop the zenith correction function, that is, an OFF command through the input device 204, the display viewpoint correction unit 25 does not have to perform the zenith correction.

Further, in the imaging system according to the second embodiment, the display viewpoint correction unit 25 of the terminal 20 performs imaging based on the orientation of the imaging unit 11A and the display direction (line Da designated by the user) designated by the image. The image captured by the unit 11A may be corrected so as to be displayed in the display direction. As a result, the terminal 20 can display the image projected in the designated display direction even when the orientation of the image capturing unit 11A changes, and can project the image in a desired orientation regardless of the orientation of the image capturing unit 11A.

(Other embodiments)
Although the example of the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment. That is, various modifications and improvements are possible within the scope of the present invention. For example, various modifications of the embodiment and forms constructed by combining the constituent elements of different embodiments are also included in the scope of the present invention.

For example, in the imaging system according to the embodiment, the display viewpoint correction unit 25 of the terminal 20 causes the position of the designated area T or the point CPa, which is an example of the display portion designated in the image, to change in the posture of the imaging device 10. However, the spherical image frame is corrected so as to be maintained at the same position on the image, but is not limited thereto.

For example, if the designated area T or the point CPa falls within a predetermined display area on the image even if the posture of the imaging device 10 changes, the display viewpoint correction unit 25 does not correct the spherical image frame, If it does not fit, the spherical image frame may be corrected. For example, the predetermined region may be a region centered on the position of the designated region T or the point CPa designated by the user. As a result, the user can visually recognize the designated area T or the point CPa without significantly changing the viewpoint.

Alternatively, the display viewpoint correction unit 25 changes the spherical image frame so as to maintain the position of the designated area T or the point CPa at a predetermined position such as the center of the image even when the posture of the imaging device 10 changes. You may correct. The center of the image may be the position of the center of the optical axis.

Further, in the image pickup system according to the embodiment, the display viewpoint correction unit 25 of the terminal 20 displays the designated display portion such as the designated area T or the point CPa on the image due to the change in the attitude of the image pickup unit 11A. The image captured by the image capturing unit 11A may be corrected so as to suppress the swing of the image.

For example, the posture data acquisition unit 24 of the terminal 20 uses the posture data of the imaging device 10 to change the direction in which the posture of the imaging device 10 changes (hereinafter, also referred to as “posture change period”) and change the posture in each direction. The amount of change in the posture of the image pickup apparatus 10 may be detected. Then, when the posture change cycle is less than or equal to the first threshold and the amount of change in posture in the cycle is less than or equal to the second threshold, the display viewpoint correction unit 25 displays the image so as to maintain the position of the display portion in the image. You may correct. In this case, it can be considered that the imaging device 10 is vibrating due to the shaking of the user who holds the imaging device 10. Further, the display viewpoint correction unit 25 may not correct the image when the posture change cycle is greater than the first threshold and the change amount of the posture in the cycle is less than or equal to the second threshold. Further, the display viewpoint correction unit 25 may correct the image so as to maintain the position of the display portion in the image when the amount of change in the posture in the posture change cycle exceeds the second threshold.

According to the above configuration, the terminal 20 can project a desired display target regardless of the posture of the imaging device 10 while suppressing the screen shake caused by the shaking of the user who holds the imaging device 10.

Further, the rocking prevention function for correcting the image may be selectable so that the display viewpoint correction unit 25 suppresses the rocking of the display portion on the image due to the change of the posture of the imaging unit 11A. For example, when the user executes the rocking suppression function, that is, an ON command is input via the input device 204 of the terminal 20, the display viewpoint correction unit 25 performs the above-described correction for suppressing the rocking. Good. Furthermore, when the user inputs a command to stop the rocking suppression function, that is, an OFF command through the input device 204, the display viewpoint correction unit 25 does not have to perform the above-described correction for suppressing the rocking. ..

In addition, in the imaging system according to the embodiment, the imaging control unit 11 of the imaging device 10 combines two image frames imaged by the first imaging element 11ab and the second imaging element 11bb to form a omnidirectional image frame. Although it is generated, the present invention is not limited to this and may not be combined with the spherical image frame. For example, the imaging control unit 11 outputs the two image frames to the moving image compression unit 12 without combining them, and the moving image compression unit 12 generates two moving image format data from each of the two image frames and performs communication. It may be transmitted to the terminal 20 via the unit 15. In this case, the imaging device 10 may transmit camera information including the orientation of the optical axis center and the imaging direction set for each of the cameras 11a and 11b to the terminal 20. Then, the terminal 20 synthesizes the image frames obtained by expanding each of the two moving image data and captured at the same time or at a similar time based on the camera information or the like to generate a celestial sphere image frame. May be. Alternatively, the terminal 20 may display the two images on the display 205 without combining the image frames obtained from the two pieces of moving image format data.

Furthermore, in the image pickup system according to the embodiment, the image pickup control unit 11 of the image pickup apparatus 10 synthesizes two image frames picked up by the first image pickup element 11ab and the second image pickup element 11bb to generate a spherical image frame. However, the present invention is not limited to this and may not be combined with the spherical image frame. For example, the imaging control unit 11 outputs the two image frames to the moving image compression unit 12 as a so-called dual fisheye image, and the moving image compression unit 12 generates the moving image format data of the dual fisheye image and performs communication. It may be transmitted to the terminal 20 via the unit 15. In this case, the imaging device 10 may transmit camera information including the orientation of the optical axis center and the imaging direction set for each of the cameras 11a and 11b to the terminal 20. Then, the terminal 20 synthesizes the image frames obtained by expanding the data of the dual fisheye moving image format and captured at the same time or at an approximate time based on the camera information and the like to generate a spherical image frame. You may. Alternatively, the terminal 20 may display the two images on the display 205 without combining the image frames obtained from the two pieces of moving image format data.

Further, in the imaging system according to the embodiment, the image captured by the imaging device 10 is transmitted to the terminal 20 via the communication network 30, but the image is not limited to this. The image transmission from the imaging device 10 to the terminal 20 may be performed via any wired communication or wireless communication. Further, the image may be stored in the storage medium from the imaging device 10, and the terminal 20 may read the image from the storage medium so that the terminal 20 acquires the image. The storage medium may be a non-volatile storage medium such as a memory card and a flash memory.

Further, in the imaging system according to the embodiment, the functions of the display image generation unit 23 and the display viewpoint correction unit 25 of the terminal 20 may be included in the imaging device 10. The imaging device 10 may perform the same processing as the display image generation unit 23 and the display viewpoint correction unit 25 before the processing in the moving image compression unit 12. At this time, the corrected image is transmitted from the imaging device 10 to the terminal 20.

Further, the imaging system according to the embodiment may include a server device connected to the communication network 30. The imaging device 10 may transmit the image to the server device, and the server device may simultaneously deliver, that is, broadcast the image to one or more terminals 20 connected to the communication network 30. In this case, the functions of the display image generation unit 23 and the display viewpoint correction unit 25 of the terminal 20 may be provided in any of the imaging device 10, the server device, and each terminal 20.

When provided in the server device, the command for the display portion designated in each terminal 20 may be transmitted from the terminal 20 to the server device. The server device performs the same processing as the display image generation unit 23 and the display viewpoint correction unit 25 to generate a corrected image for each display portion, and instructs each terminal 20 to display the instructed display portion. The corrected image corresponding to may be distributed. Alternatively, the server device may generate an image in which the corrected images of the respective display portions are aggregated and deliver the aggregated image to each terminal 20. In this case, one image includes the images of all display parts.

Further, when provided in the imaging device 10, the command for the display portion of each terminal 20 may be transmitted from the terminal 20 to the server device. The server device may collect the information of the display portion of each terminal 20 and transmit the information to the imaging device 10. The imaging device 10 may generate the corrected image for each display portion by performing the same processing as the display image generation unit 23 and the display viewpoint correction unit 25, and may transmit the corrected image to the server device. Then, the server device may deliver the corrected image corresponding to the instructed display portion to each terminal 20. Alternatively, the imaging device 10 may generate an image in which the corrected images of the respective display portions are aggregated and transmit the image to the server device.

Further, when provided in each terminal 20, the image captured by the imaging device 10 may be transmitted to each terminal 20 via the server device. Each terminal 20 may generate a corrected image of the display portion designated on the terminal 20.

Further, in the terminal 20 of the imaging system according to the embodiment, the user specifies one display portion via the input device 204, and the display viewpoint correction unit 25 maintains the position of the one display portion. The image has been corrected, but is not limited to this. For example, a plurality of display parts may be designated via the input device 204. In this case, the display viewpoint correction unit 25 may generate, for each of the plurality of display portions, a plurality of corrected images in which the position of the display portion is maintained. The display control unit 26 may cause the display 205 to sequentially display a plurality of corrected images, or may display at least two of the plurality of corrected images at the same time.

Further, the imaging device 10 may be equipped with a GPS (Global Positioning System). When the GPS is provided, the GPS information is incorporated in the metadata of the image data to be transmitted to the terminal 20, and the terminal 20 displays the image data and the map in association with the separately provided map information. The above position may be displayed in conjunction with each other. With this configuration, the viewer can also confirm the position of the imaging device 10.

Further, the present invention may be a program or a non-transitory computer-readable recording medium in which the program is recorded. Further, it goes without saying that the above program can be distributed via a transmission medium such as the Internet.

For example, a program according to an embodiment of the present invention is a program to be executed by a computer of a terminal, and includes a process of acquiring an image captured by an image capturing unit and information about the orientation of the image capturing unit, And a process of correcting the acquired image based on the display part designated by the image so as to project the display part, and a process of displaying the corrected image. According to this program, the same effect as the terminal 20 can be obtained.

Further, in the correction processing, the program projects the acquired image in the display direction from the acquired image based on the attitude of the imaging unit and the display portion and the display direction specified in the image. May be corrected as follows.

Further, the program corrects the acquired image in the correction process so that the vertical axis of the acquired image is the zenith direction, and the vertical axis is set in the imaging unit. It may correspond to the vertical axis.

Further, the program may correct the acquired image in the correction process so as to suppress the swing of the display portion on the image due to the change in the attitude of the imaging unit.

Further, the program may correct the acquired image in the correction process so as to maintain the position of the display portion with respect to the image regardless of the change in the posture of the imaging unit.

Further, in the above program, the attitude of the imaging unit may include attitudes of the imaging unit in the yawing, pitching, and rolling directions.

Moreover, all the numbers such as ordinal numbers and quantities used above are examples for specifically explaining the technique of the present invention, and the present invention is not limited to the exemplified numbers. Further, the connection relationship between the constituent elements is an example for specifically explaining the technique of the present invention, and the connection relationship for realizing the function of the present invention is not limited to this.

Further, the block division in the functional block diagram is an example, and even if a plurality of blocks are realized as one block, one block is divided into a plurality of blocks, and/or a part of the functions is transferred to another block. Good. Also, the functions of a plurality of blocks having similar functions may be processed in parallel or in time division by a single piece of hardware or software.

10 Imaging Device 11A Imaging Units 11a and 11b Camera 12 Video Compression Unit (Example of Generation Unit)
13 Attitude Detection Unit 15 Communication Unit (Example of Transmission Unit)
20 terminal 21 communication unit (an example of acquisition unit)
22 Video expansion unit (an example of expansion unit)
25 Display viewpoint correction unit (an example of correction unit)
26 Display Control Unit (Example of Display Unit)
30 communication network 1000 imaging system

Japanese Patent No. 5843033

Claims (19)

An image capturing unit for capturing an image,
A detection unit that detects the attitude of the imaging unit;
A correction unit that corrects the image captured by the image capturing unit based on the attitude of the image capturing unit and the display unit designated in the image so that the display unit is displayed.
An image pickup system, comprising: a display unit that displays the corrected image. The correction unit corrects the image captured by the image capturing unit based on the attitude of the image capturing unit and the display portion and the display direction specified in the image so that the display portion is projected in the display direction. The imaging system according to claim 1. The correction unit corrects the image captured by the image capturing unit such that the vertical axis of the image captured by the image capturing unit is the zenith direction,
The imaging system according to claim 2, wherein the vertical axis corresponds to a vertical axis set in the imaging unit. The correction unit corrects the image captured by the image capturing unit so as to suppress the swinging of the display portion on the image due to a change in the attitude of the image capturing unit. The imaging system described. The correction unit corrects the image captured by the imaging unit so as to maintain the position of the display portion with respect to the image regardless of a change in the attitude of the imaging unit. The imaging system described. The imaging system according to any one of claims 1 to 5, wherein the attitude of the imaging unit includes attitudes of the imaging unit in yawing, pitching, and rolling directions. An imaging device and a terminal are provided,
The imaging device is
The imaging unit;
A generating unit that generates moving image data from a plurality of images captured by the image capturing unit;
The detection unit;
A transmitting unit that sends the moving image data and the attitude of the imaging unit to the terminal,
The terminal is
A development unit that develops the moving image data into individual images,
The correction unit,
The imaging system according to claim 1, further comprising: the display unit. An acquisition unit that acquires an image captured by the image capturing unit and information about the orientation of the image capturing unit;
A correction unit that corrects the image acquired by the acquisition unit based on the attitude of the image capturing unit and the display portion specified in the image so that the display portion is projected.
A terminal that includes a display unit that displays the corrected image. The correction unit corrects the image acquired by the acquisition unit based on the attitude of the imaging unit and the display portion and the display direction specified in the image so that the display portion is projected in the display direction. The terminal according to claim 8. The correction unit corrects the image acquired by the acquisition unit such that the vertical axis of the image acquired by the acquisition unit is the zenith direction,
The terminal according to claim 9, wherein the vertical axis corresponds to the vertical axis set in the imaging unit. The correction unit corrects the image acquired by the acquisition unit so as to suppress rocking of the display portion on the image due to a change in the attitude of the imaging unit. The listed terminal. The correction unit corrects the image acquired by the acquisition unit so as to maintain the position of the display portion with respect to the image regardless of a change in the posture of the imaging unit. The terminal described in. The terminal according to any one of claims 8 to 12, wherein the attitude of the imaging unit includes attitudes of the imaging unit in the yawing, pitching, and rolling directions. A program to be executed by a terminal computer,
A process of acquiring an image captured by the image capturing unit and information on the attitude of the image capturing unit;
A process of correcting the acquired image based on a posture of the image capturing unit and a display portion designated in the image so as to project the display portion;
A program including a process of displaying the corrected image. In the correction processing, the acquired image is corrected based on the attitude of the imaging unit and the display portion and the display direction specified in the image so that the display portion is projected in the display direction. 14. The program according to 14. In the correction process, the acquired image is corrected so that the vertical axis of the acquired image is the zenith direction,
The program according to claim 15, wherein the vertical axis corresponds to a vertical axis set in the imaging unit. The correction process corrects the acquired image so as to suppress the swing of the display portion on the image due to the change in the attitude of the imaging unit. program. The processing for correcting corrects the acquired image so as to maintain the position of the display portion with respect to the image regardless of the change in the attitude of the imaging unit. Program of. The program according to any one of claims 14 to 18, wherein the attitude of the imaging unit includes attitudes of yawing, pitching, and rolling of the imaging unit.

Images