JP6741498B2 - Imaging device, display device, and imaging display system - Google Patents

Description

The present invention relates to a technique of an image pickup device and a display device, and relates to a technique of displaying a captured image on a screen.

2. Description of the Related Art Conventionally, there are digital cameras, smartphones, and the like as electronic devices such as an imaging device having a function of capturing a plurality of images with different shooting directions and angles of view. Further, as an electronic device such as a display device having a function of reproducing and displaying the plurality of images, there are a television receiver (hereinafter sometimes referred to as “TV”), a notebook PC, and the like. The image is a still image or a moving image. The shooting direction can be defined by, for example, an azimuth angle and an elevation angle. The angle of view represents an image range by an angle and is also called a viewing angle, and can be defined by, for example, a horizontal angle of view corresponding to an azimuth angle and a vertical angle of view corresponding to an elevation angle.

Images with different angles of view include images that are taken relatively wide-angle (hereinafter sometimes referred to as "wide-angle image") and images that are taken relatively narrow-angle (hereinafter "narrow-angle"). It may be described as "image" and the like). Examples of the wide angle include a horizontal angle of view of 120° and an angle of view of 180° or more. It should be noted that the case of 180° may be referred to as a hemispherical image, and the case of 360° may be referred to as a spherical image. An example of the narrow angle is a horizontal angle of view of 50° or less. Hereinafter, the unit of the angle is expressed in degrees (°), but it can be converted into a unit such as radian.

Japanese Patent Application Laid-Open No. 2000-92439 (Patent Document 1) can be cited as a prior art example relating to screen display of a captured image. Patent Document 1 describes the following as an electronic still camera and the like. The electronic still camera, when instructed to turn off the power, generates an image list display file that enables the external device to display a list of the recorded images recorded in the recording memory. A plurality of reduced images are displayed in a list on the list display screen displayed by the image search using the file.

JP-A-2000-92439

A user who is a photographer may use an image capturing device to capture a plurality of images having different image capturing directions and different angles of view at the same place and time. For example, a user shoots a wide-angle image in a certain shooting direction and angle of view and a narrow-angle image in another shooting direction and angle of view in the same park on the same day. In such a case, it can be said that the plurality of images are related images having a predetermined relationship with respect to place, date and time, shooting direction, angle of view, and the like.

Conventional imaging devices and display devices have a function of displaying a plurality of images on a display screen in a list and in a side-by-side arrangement, for example, in the order of shooting date, file name, and the like. For example, in the case of Patent Document 1, a plurality of reduced images are displayed vertically and horizontally side by side on a list display screen.

However, in the conventional technology, it is difficult for the user to intuitively recognize a plurality of images having a predetermined relationship among all the images displayed on the display screen. For example, even when a plurality of images including a wide-angle image and a narrow-angle image captured at the same place and date and time are mixed, it is difficult for the user to grasp the mutual relationship of the plurality of images on the list display screen. .. The related art does not have a function of reproducing and displaying a plurality of related images in association with each other. The related art does not have a function of displaying an image on the display screen and displaying it by referring to another image related to the image. In the conventional technology, it takes a lot of time and labor for the user to edit and manage a plurality of images.

That is, the related art has difficulty in suitably viewing a plurality of related images, and there is room for improvement in terms of usability.

It is an object of the present invention to provide a technique for an imaging device and a display device, which makes it easy to understand the relationship between a plurality of images and realizes better usability.

A typical embodiment of the present invention is an image pickup device or the like, which is characterized by having the following configuration.

An image pickup apparatus according to an embodiment includes an image pickup unit that obtains image data by picking up a still image or a moving image, and a display unit that reproduces and displays the image on a display screen based on the image data. The first information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of the angle of view of the image and the angle-of-view related information capable of calculating the angle of view is acquired and imaged. For a plurality of images, the second shooting direction or partial view angle of the second image is included within the first view angle range of the first image or within the threshold view angle range extended outside thereof. If the second image is associated with the first image, the first image selected based on a user operation or the first figure representing the first image is displayed on the background area in the display screen. And a second graphic representing the second image or the second image associated with the first image in a positional relationship corresponding to the shooting direction.

According to the representative embodiment of the present invention, it is possible to easily understand the relationship between a plurality of images in the imaging device and the display device, and to realize better usability.

It is a figure which shows the structure of the imaging display system comprised including the imaging device and display device of Embodiment 1 of this invention. FIG. 3 is a diagram showing a configuration of the image pickup apparatus according to the first embodiment. FIG. 3 is a diagram showing a configuration of the display device according to the first embodiment. FIG. 3 is a diagram showing an appearance of the image pickup apparatus according to the first embodiment. 3 is a diagram showing a configuration of management information and metadata in the image pickup apparatus according to the first embodiment. FIG. 5 is a diagram illustrating management of a relationship between a plurality of images in the image pickup apparatus according to the first embodiment. FIG. FIG. 3 is an explanatory diagram showing a shooting direction and an angle of view regarding a plurality of images, particularly an azimuth angle and a horizontal angle of view, in the image pickup apparatus according to the first embodiment. FIG. 3 is an explanatory diagram particularly showing an elevation angle and a vertical angle of view in the image pickup apparatus according to the first embodiment. FIG. 4 is an explanatory diagram showing angle-of-view related information in the image pickup apparatus according to the first embodiment. FIG. 5 is a diagram showing a transition of a display mode of a display screen in the image pickup apparatus according to the first embodiment. FIG. 11 is a diagram showing an example of a second display mode screen in the image pickup apparatus of the modified example of the first embodiment. FIG. 11 is a diagram showing an example of a second display mode screen in the image pickup apparatus of the modified example of the first embodiment. FIG. 8 is a diagram showing another image example of a second display mode screen in the image pickup apparatus according to the first embodiment. FIG. 8 is a diagram showing another image example of a second display mode screen in the image pickup apparatus according to the first embodiment. FIG. 8 is a diagram showing another image example of a second display mode screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of scrolling a second display mode screen and the like in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of enlarging/reducing a second display mode screen and the like in the imaging device of the first embodiment. FIG. 6 is a diagram showing an example of a second display mode screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of a second display mode screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of a fourth display mode screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing an example of a map display mode screen in the image pickup apparatus according to the first embodiment. FIG. 6 is a diagram showing a flow of control processing at the time of image capturing in the image capturing apparatus of the first embodiment. FIG. 4 is a diagram showing a flow of control processing at the time of reproduction display in the image pickup apparatus and the display apparatus according to the first embodiment. 4 is an explanatory diagram showing a first selection method in the image pickup apparatus according to the first embodiment. FIG. FIG. 4 is an explanatory diagram showing a second selection method in the image pickup apparatus according to the first embodiment. FIG. 6 is an explanatory diagram showing a third selection method in the image pickup apparatus according to the first embodiment. FIG. 4 is an explanatory diagram showing association conditions and the like in the image pickup apparatus according to the first embodiment. FIG. 11 is a diagram showing an example of a second display mode screen in the image pickup apparatus of the modified example of the first embodiment. FIG. 11 is a diagram showing an example of a second display mode screen in the image pickup apparatus of the modified example of the first embodiment. It is explanatory drawing which shows the imaging mode in the imaging device of Embodiment 2 of this invention. FIG. 11 is a diagram showing a shooting menu screen in the image pickup apparatus according to the second embodiment. FIG. 16 is a diagram showing a display screen of a monitor image in a shooting mode in the image pickup apparatus according to the third embodiment of the present invention. FIG. 16 is a diagram showing a display screen of a monitor image in a shooting mode in the image pickup apparatus of the modified example of the third embodiment. It is a figure which shows the example of a display screen of a 1st edit function in the imaging device of Embodiment 4 of this invention. FIG. 16 is a diagram showing an example of a display screen of a second editing function in the image pickup apparatus according to the fourth embodiment. It is explanatory drawing shown about the omnidirectional image in the imaging device of Embodiment 5 of this invention. FIG. 16 is a diagram showing a configuration relating to an image pickup unit in the image pickup apparatus according to the fifth embodiment. FIG. 16 is a diagram showing an external configuration of the image pickup apparatus according to the fifth embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In all the drawings for explaining the embodiments, the same parts are designated by the same reference numerals in principle, and the repeated description thereof will be omitted.

(Embodiment 1)
The imaging device and the display device according to the first embodiment of the present invention will be described with reference to FIGS. The image pickup apparatus and the display apparatus according to the first embodiment have a function of displaying a plurality of images having different shooting directions and field angles at the time of shooting in association with each other.

[Imaging display system]
FIG. 1 shows a configuration of an image pickup display system which is a system including the image pickup apparatus and the display apparatus according to the first embodiment. The image pickup display system of FIG. 1 includes a first image pickup apparatus 1A and a second image pickup apparatus 1B which are the image pickup apparatus 1 of the first embodiment, and a first display apparatus 2A and a second display which are the display apparatus 2 of the first embodiment. The device 2B and the server 3 are connected via the communication network 4. The communication network 4 includes a wireless communication network and the Internet.

The image pickup apparatus 1 is an electronic device having at least an image pickup function and a display function. The display device 2 is an electronic device having at least a display function. A user who is a photographer has a first imaging device 1A and a second imaging device 1B, and a first display device 2A and a second display device 2B as electronic devices that the user owns or has authority to use. For example, the first imaging device 1A is a digital camera, the second imaging device 1B is a smartphone, the first display device 2A is a television, and the second display device 2B is a notebook PC.

The first image capturing apparatus 1A and the second image capturing apparatus 1B have at least a function of holding image data of an image captured by the own device in a storage unit in the own device. In addition, the first imaging device 1A and the second imaging device 1B may have a function of transmitting image data of an image captured by the self device to the server 3 and accumulating and registering the image data in the image data DB of the server 3. .. In addition, the first image pickup device 1A and the second image pickup device 1B have a function of transmitting image data of an image picked up by the device itself to the first display device 2A and the second display device 2B based on a user operation. May be.

The first display device 2A and the second display device 2B have a function of receiving and inputting image data from the external device such as the imaging device 1 or the server 3 and storing the image data in a storage unit in the device itself. For example, the first display device 2A receives image data from the first imaging device 1A and the second display device 2B by communication. For example, the second display device 2B receives image data from the first imaging device 1A and the second imaging device 1B by communication. In addition, the first display device 2A and the second display device 2B have an image data display unit. The image data display unit displays an image on the display screen based on the image data. In addition, for example, the second display device 2B has an image data editing unit. The image data editing unit edits information and contents of image data based on a user operation on the display screen.

The server 3 is, for example, a server device managed by a company, stores image data from a user's device, and stores the image data in an image data DB. This realizes backup of image data, unified management, and the like. The server 3 and the image data DB may be implemented by different devices.

[Imaging device]
FIG. 2 shows the configuration of the image pickup apparatus 1. The imaging device 1 includes a control unit 101, a storage unit 102, an imaging unit 103, a display device 104, an operation input unit 105, a recording unit 106, a recording medium 107, a communication interface unit 108, a photographing direction measuring unit 111, an internal clock 112, and a position. The detection unit 113, the sensor group 114, the microphone 115, the speaker 116, the interface circuit unit 131, the signal processing unit 132, the memory 133 and the like are provided. These units are connected to each other via a bus or the like, and cooperate with each other at high speed under the control of the control unit 101.

The control unit 101 controls the entire imaging device 1. The control unit 101 includes a processor and control circuits such as a ROM and a RAM. The control unit 101 includes an imaging control unit 11, a display control unit 12, a touch detection unit 13, a setting unit 14, and an association unit 15 as processing units implemented by the processing of the control circuit.

The control unit 101 controls each unit such as the image pickup unit 103 according to the operation mode to realize image pickup and display. The operation mode includes an image pickup mode, a reproduction display mode, and the like. The control unit 101 controls the imaging unit 103, the display device 104, the recording unit 106, and the like via the signal processing unit 132 and the memory 133 according to the operation mode and state.

The control unit 101 controls settings such as a shooting direction and an angle of view of imaging based on a user input operation through the operation input unit 105 and the touch sensor 42. The control unit 101 creates the metadata 22 together with the image data 23 as the image is captured by the imaging unit 103, and stores the image data 24 including the metadata 22 and the image data 23 in the storage unit 102 or the recording unit. It records on the recording medium 107 of 106. Further, the control unit 101 creates and manages the management information 21 regarding the image file 24.

The storage unit 102 stores various data and information including the plurality of image files 24 and the management information 21 under the control of the control unit 101. Note that the storage unit 102 may be integrated in the control unit 101, the storage unit 102 and the recording unit 106 may be integrated in one form, or the like.

The image capturing unit 103 captures an image under the control of the image capturing control unit 11. The imaging unit 103 includes a drive unit 31 and a lens unit 32. The drive unit 31 includes a drive circuit and drives the image pickup device and the lens of the lens unit 32 based on the drive control from the image pickup control unit 11. The lens unit 32 includes an optical system including a plurality of lenses including a lens compatible with electronic zoom or optical zoom. For example, the driving unit 31 performs a focusing operation that changes the focal length by controlling the position of the focus lens. The lens of the lens unit 32 may be a zoom lens system that is fixed without replacement and is compatible with electronic zoom, or may be an optical lens system that can be replaced by being detached.

The image pickup unit 103 includes an image pickup device such as a CMOS or CCD. Photoelectric conversion elements are two-dimensionally arranged on the image pickup surface of the image pickup element. The image capturing unit 103 photoelectrically converts the optical image of the subject, which is incident through the lens unit 32 and formed on the image capturing surface of the image sensor, into an image capturing signal. The image pickup unit 103 has a built-in AD conversion circuit that converts an analog signal into a digital signal, and outputs a digitized image pickup signal. The image pickup unit 103 may use an image pickup device in which pixels for phase difference autofocus are arranged to speed up autofocus, or may have a built-in memory to speed up input/output of an image pickup signal. Good. An AD conversion circuit or the like may be provided outside the image capturing unit 103. The imaging method of the imaging unit 103 is not limited.

The display device 104 is, for example, a touch panel, and includes a display unit 41 and a touch sensor 42. The touch panel is a display unit and an operation input unit. The display unit 41 includes a display driver and displays a display screen to the user. The display unit 41 is, for example, a liquid crystal display unit, but may be an organic EL display unit or the like. The method of the display unit 41 and the method of the touch sensor 42 are not limited. A plurality of display devices 104 may be provided, or the display device 104 may not be provided with the touch sensor 42. In the first embodiment, in addition to the touch panel that is the main display device 104, a display device for a photographing monitor, which will be described later, is provided.

The touch sensor 42 is arranged corresponding to the area of the display screen of the display unit 41, and receives a touch input operation on the display screen. The touch sensor 42 is, for example, a capacitance type, detects capacitance change due to proximity or contact of a finger or the like as an electric signal, and outputs a touch detection signal to the touch detection unit 13 of the control unit 101. In the touch panel, for example, a touch sensor 42 is built in a liquid crystal display unit, a glass protective cover is arranged on the front side, and a backlight is arranged on the back side. Note that the display device 104 may be automatically controlled to be in the non-display state by turning off the backlight based on the non-use state for a certain period of time. The touch panel may be a pressure sensitive type or the like.

The touch detection unit 13 detects the presence/absence of a touch on the display screen, the touch position coordinates, and the like based on the touch detection signal obtained from the touch sensor 42, and also detects operations such as taps, swipes, and pinches. By a touch input operation on the display screen of the touch panel, the user can perform various operations such as an instruction input to the imaging device 1 and an operation of focusing on an arbitrary position on the display screen.

The interface circuit unit 131 transfers the drive control signal from the control unit 101 to the imaging unit 103 under the control of the control unit 101 according to the operation mode. Further, the interface circuit unit 131 outputs the image pickup signal from the image pickup unit 103 to the signal processing unit 132 or the memory 133 under control. The interface circuit unit 131 may be connected to a plurality of image capturing units 103.

The signal processing unit 132 includes an image/audio signal processing circuit and an encoding/decoding circuit, and under the control of the control unit 101, various signal processing such as image signal processing, audio signal processing, encoding processing, and decoding processing. Has the function of performing. The signal processing of the signal processing unit 132 includes signal processing for the image pickup signal from the image pickup unit 103, signal processing for the image data 23, and signal processing for displaying an image on the display device 104. The signal processing unit 132 generates a video signal for recording or display by signal processing. This video signal includes a still image or a moving image of the image data 23.

The signal processing unit 132 performs signal processing such as filtering, amplification according to sensitivity setting, and white balance correction on the image pickup signal, for example. When recording the image data 23 as the image file 24 on the recording medium 107, the signal processing unit 132 performs a predetermined format encoding process for the original data. This encoding includes data compression. When reading the image data 23 from the recording medium 107, the signal processing unit 132 performs a corresponding decoding process to restore the original data. The signal processing unit 132 performs signal processing for reproducing and displaying an image on the display device 104 based on the image data 23. The signal processing unit 132 performs audio signal processing on an audio input signal from the microphone 115 and an audio output signal to the speaker 116.

The memory 133 is used as a buffer memory at the time of signal processing of the signal processing unit 132, and stores each signal and data. The memory 133 is composed of, for example, a DRAM or a flash memory.

The operation input unit 105 includes various hardware buttons and the like that enable an input operation by the user. The user can input an instruction through the operation input unit 105 and can set the view angle related information and the like in the setting unit 14.

The setting unit 14 accepts settings input by the user through the operation input unit 105 and the touch panel, and sets information relating to shooting and display. The setting unit 14 can set the electronic zoom magnification, the focal length, the angle of view, and the like regarding the lens of the image capturing unit 103 as the setting information related to shooting. The imaging control unit 11 drives and controls the imaging unit 103 based on the setting information of the setting unit 14 so as to automatically adjust the angle of view and the like.

The recording unit 106 includes a recording/reproducing circuit and the like, and a recording medium 107 is detachably provided. The recording medium 107 is, for example, an SD card. The recording unit 106 records the image file 24 including the image data 23 in the storage unit 102 or the memory 133 on the recording medium 107 under the control of the control unit 101. Under the control of the control unit 101, the recording unit 106 reads the image file 24 from the recording medium 107 and stores it in the storage unit 102, the memory 133, or the like. The user can remove the recording medium 107 and carry or replace it. The image file 24 may include audio data together with the image data 23.

The communication interface unit 108 includes a wireless communication interface unit. Under the control of the control unit 101, the communication interface unit 108 connects to the communication network 4 via a wireless access point or the like, and performs communication processing with an external device. For example, as illustrated in FIG. 1, the imaging device 1 communicates with the server 3 or the display device 2 or the imaging device 1 other than the own device via the communication network 4, and transmits the image file 24 of the own device, or an external device. The image file 24 is received from the device. At that time, the communication interface unit 108 transfers the communication data of the image file 24. The communication interface unit 108 may have a telephone network communication function, a LAN communication function, a near field communication function, and the like.

The shooting direction measurement unit 111 includes sensors such as an electronic compass, a gyro sensor, and an acceleration sensor, and uses them to measure the shooting direction of the imaging device 1 and output information on the shooting direction to the control unit 101. The shooting direction measurement unit 111 measures an azimuth angle and an elevation angle as a shooting direction. Further, the shooting direction measurement unit 111 detects the rotation angle of the image at the time of shooting based on the detection of the attitude of the imaging device 1. The rotation angle of the image is represented by the angle formed by the horizontal and vertical lines of the rectangle of the image with respect to the horizontal and vertical directions.

The internal clock 112 measures the current date and time. Accordingly, the control unit 101 can obtain the shooting date and time. The position detection unit 113 includes a GPS reception unit and detects the position of the imaging device 1 using a signal from the satellite. The position detection unit 113 detects {latitude, longitude, altitude (altitude above sea level)} as position information. The altitude may be omitted. The sensor group 114 is another sensor group, and examples thereof include a proximity sensor and an illuminance sensor. The microphone 115 is a voice input device, and the speaker 116 is a voice output device.

The image pickup control unit 11 controls the image pickup unit 103 and the like at the time of image pickup in accordance with the operation mode of the image pickup apparatus 1 to pick up an image and obtain image data 23. The imaging control unit 11 stores the image data 23 as an image file 24 in the storage unit 102 or the like. The image capturing unit 103 and the image capturing control unit 11 have a function of enabling capturing of both wide-angle images and narrow-angle images. The imaging control unit 11 performs zoom control regarding the optical system of the imaging unit 103. The zoom control is control of electronic zoom or optical zoom. The imaging control unit 11 grasps and adjusts the electronic zoom magnification, the focal length, the angle of view, etc. regarding the lens of the imaging unit 103 based on the setting information such as the angle-of-view related information of the setting unit 14.

In the case where the imaging unit 103 can replace the optical lens with a different focal length or the like, for example, when capturing a narrow-angle image, the user replaces the optical lens with a relatively long focal length and captures a wide-angle image. In that case, replace it with an optical lens having a relatively short focal length. When the user replaces the lens of the image capturing unit 103, the setting unit 14 may automatically detect the replacement and detect the focal length of the lens after the replacement. Alternatively, after the replacement, the user may input and set the focal length and the like of the replaced lens into the setting unit 14. Then, the imaging control unit 11 acquires the updated setting information from the setting unit 14 and grasps the focal length and the like.

The display control unit 12 controls the display unit 41 and the like of the display device 104 to display images and information on the display screen of the display unit 41. The display control unit 12 displays the folder information and the image on the display screen based on the management information 21 and the image file 24. The display control unit 12 provides various screens that serve as a graphical user interface for the user. The display control unit 12 controls screen switching according to the reproduction display mode. The display control unit 12 also provides a menu screen, a setting screen, and the like.

The associating unit 15 cooperates with the image capturing control unit 11 and the display control unit 12 to perform an associating process of associating a plurality of images having a predetermined relationship. The associating unit 15 performs the associating process according to an associating method described later at a predetermined timing and timing such as at the time of image capturing or reproduction display. As a first process in the association process, the associating unit 15 determines a plurality of image data 23 in the storage unit 102 or the like for association. In the determination of the first process, an image that is a candidate for association is selected based on conditions such as a shooting location according to a selection method described later. As a second process, the associating unit 15 determines a first image and a second image to be associated with each other in a plurality of candidate images based on the conditions of the shooting direction and the angle of view. As a result of the determination, the associating unit 15 creates the metadata 22 or the management information 21 so that the first image and the second image are associated with each other.

When the image is reproduced and displayed, the display control unit 12 has a predetermined relationship based on the determination result of the associating unit 15 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24. The first image and the second image are specified. The display control unit 12 displays a plurality of images including the first image and the second image in a state associated with each other on the display screen.

The hardware configuration of each processing unit and each function such as the control unit 101 and the signal processing unit 132 of the image pickup apparatus 1 may be a form in which each processing unit and each function are implemented by a circuit such as an individual LSI. It may be a form implemented by one integrated circuit such as an LSI.

[Display device]
FIG. 3 shows the configuration of the display device 2. The display device 2 does not include the image capturing unit 103 and a processing unit related to image capturing, which is a configuration mainly different from the image capturing device 1. Of the components of the display device 2, those similar to the components of the imaging device 1 will not be described. The display device 2 includes a control unit 201, a storage unit 202, a display unit 204, an operation input unit 205, a recording unit 206, a recording medium 207, a communication interface unit 208, a sensor group, a microphone, a speaker, and the like. These units are connected to each other via a bus or the like, and cooperate with each other at high speed under the control of the control unit 201.

The control unit 201 controls the entire display device 2. The control unit 201 has a display control unit 212 and an associating unit 215 as processing units realized by the processing of the control circuit. The control unit 201 controls the display on the display screen of the display unit 204 based on the user input operation through the operation input unit 205.

The storage unit 202 stores various data and information including the plurality of image files 24 and the management information 21 under the control of the control unit 201. The image file 24 and the management information 21 may be created by the own device or acquired from an external device.

The display unit 204 includes a display driver and displays a display screen to the user. The display unit 204 may be a touch panel.

The operation input unit 205 includes various hardware buttons and the like that allow an input operation by the user. The operation input unit 205 may include a remote controller and a remote controller light receiving unit. The user can input instructions and make user settings through the operation input unit 205.

The recording unit 206 includes a recording/reproducing circuit and the like, and a recording medium 207 is detachably provided. The recording unit 206 records the image file 24 including the image data 23 of the storage unit 202 in the recording medium 207 under the control of the control unit 201. The recording unit 206 reads the image file 24 from the recording medium 207 and stores it in the storage unit 202 under the control of the control unit 201.

The communication interface unit 208 connects to the communication network 4 under the control of the control unit 201 and performs communication processing with an external device. For example, as shown in FIG. 1, the display device 2 communicates with the server 3 or the imaging device 1 or the display device 2 other than the own device via the communication network 4, and transmits the image file 24 of the own device, or an external device. The image file 24 is received from the device. At that time, the communication interface unit 208 transfers the communication data of the image file 24.

The display control unit 212 controls the display unit 204 and the like to display an image and information on the display screen of the display unit 204. The display control unit 212 reproduces and displays the folder information and the image on the display screen of the display unit 204 based on the image file 24 and the management information 21. The display control unit 212 controls screen switching according to the reproduction display mode.

The associating unit 215 cooperates with the display control unit 212 to perform an associating process for associating a plurality of images having a predetermined relationship. The associating unit 215 can also perform the associating process on the image file 24 acquired from the external device. The associating process of the associating unit 215 is similar to the associating process of the associating unit 15.

The display control unit 212 has a predetermined relationship based on the determination result of the associating unit 215 or the reference confirmation of the metadata 22 or the management information 21 of the associated image file 24 when the image is reproduced and displayed. The first image and the second image are specified. The display control unit 212 displays a plurality of images including the first image and the second image in a state of being associated with each other on the display screen.

Hereinafter, the image pickup apparatus 1 will be mainly described, but the functions other than the functions and operations related to image pickup can be realized mainly by the display apparatus 2.

[Imaging device-appearance]
FIG. 4 shows an example of the external configuration of the image pickup apparatus 1. The left side of FIG. 4 shows the configuration of the image pickup apparatus 1 seen from the back side of the housing 400, and the right side shows the configuration of the right side of the housing 400. The back surface is the surface of the display unit 41 on which the display screen 401 is provided, and the front surface is the surface of the imaging unit 103 on which the lens 402 is provided.

In the side surface configuration, the lens 402 of the lens unit 32 is arranged at the front center position of the housing 400. Further, an electronic viewfinder (EVF) 403 is provided, for example, in a state of projecting upward with respect to the main housing 400. The EVF 403 is one of the display devices 104.

In the rear structure, a rectangular display screen 401 corresponding to the touch panel described above is arranged, and information display and touch input operation are possible. The EVF 403 also has a display screen 404 and a proximity sensor 405. The EVF 403 displays an image of a subject as a monitor image based on a signal input via a lens and an image sensor at the time of shooting. The user can shoot while checking the monitor image of the EVF 403. Information such as the aperture value may be displayed in an overlapping manner on the display screen 404. The proximity sensor 405 detects the proximity of an object. The proximity sensor 405 is of a capacitance type, an optical type, or the like. For example, the control unit 101 determines whether or not the user is using the EVF 403 by detecting the presence or absence of proximity through the proximity sensor 405. The control unit 101 automatically switches between the display on the display screen 401 and the display of the monitor image on the display screen 404 of the EVF 403 according to the determination. Note that the EVF 403 is not the only option, and an optical finder may be mounted. Further, a mode for displaying a monitor image on the display screen 401 can also be set.

The housing 400 is provided with a mode setting dial 431, a shutter button 432, a cross key 433, a reproduction display mode button 434, a power button 435, and the like as the operation input unit 105. The cross key 433 includes up/down/left/right buttons and a center button, and can be used for selecting information at a desired position in the display screen 401, inputting decision, cancellation, and the like. A menu can be displayed on the display screen 401 based on a predetermined input operation.

The mode setting dial 431 is used to set the operation mode. The shutter button 432 is pressed at the time of shooting. The replay display mode button 434 is used when replaying and displaying on the display screen 401, and the replay display mode can be switched. The power button 435 is used to turn on/off the power. The user can perform various input operations such as displaying a monitor image on the display screen 401 or the display screen 404 and touching a desired position to focus.

[Imaging device-operation mode]
The outline of the operation in the shooting mode as the operation mode of the image pickup apparatus 1 is as follows. When the user presses the power button 435 to turn on the power, the normal shooting mode is activated. The control unit 101 drives and controls the imaging unit 103 and the like according to the normal shooting mode. In the normal shooting mode, it is possible to shoot a still image or a moving image as an image. At the time of shooting, the control unit 101 reads out an image pickup signal from the image pickup unit 103 in a predetermined cycle, performs predetermined signal processing in the signal processing unit 132 via the interface circuit unit 131, and converts it into a video signal in a format for display. .. The control unit 101 displays the monitor image in real time on the display screen 401 of the touch panel of the display device 104 or the display screen 404 of the EVF 403 based on the video signal. The user shoots as follows while checking the monitor image on the display screen 401 or the display screen 404.

The user sets the angle-of-view related information and the like as shooting conditions in the setting unit 14 through the operation input unit 105. When taking a still image, the user presses the shutter button 432. The imaging control unit 11 detects pressing of the shutter button 432 and determines the aperture value, shutter speed, focus, etc. according to the set shooting conditions. The shutter button 432 is, for example, half-pressed or fully-pressed. The imaging control unit 11 sets focus and the like in the half-pressed state, and controls to start imaging in the fully-pressed state.

A still image pickup signal picked up by the image pickup unit 103 is subjected to predetermined signal processing for a still image using the signal processing unit 132 and the memory 133. For example, JPEG is applied as the encoding method at that time, but MPEG, high-quality RAW format, or the like may be applied as another method. The image file 24 corresponding to the encoded still image data is recorded on the recording medium 107 by the recording unit 106, for example.

When shooting a moving image, the user presses the moving image shooting button to start moving image shooting, and presses the moving image shooting button again to stop moving image shooting. The image pickup signal of the moving image is subjected to signal processing of a predetermined method for the moving image. As the method, for example, MPEG such as H264 and H265, and other formats are applied. The encoded moving image data is similarly recorded in, for example, the recording medium 107.

The operation outline of the reproduction display mode is as follows. When the control unit 101 detects that the reproduction display mode has been set by the reproduction display mode button 434, the control unit 101 reads the image file 24 from the recording medium 107 of the recording unit 106 and stores it in the storage unit 102. The display control unit 12 uses the signal processing unit 132 to restore the original data of the image from the image data 23 by the decoding process, and generates video data for displaying the image or the like on the display screen. The display control unit 12 controls the display device 104 based on the video data and displays an image or the like on the display screen.

[Management information and image files]
FIG. 5 shows a configuration example of the management information 21 and the image file 24 of the storage unit 102. The management information 21 includes image file management information and association management information. The image file management information is information managed by a known file system, and includes, for example, file name, update date/time, type, size, and the like. The association management information is unique information created to manage the relationship between a plurality of images. The association management information includes, for example, a type identifier, a reference identifier, and a setting character string.

The type identifier is an identifier for identifying the type of image for each image file 24. There are two types of images, for example, a wide-angle image and a narrow-angle image. The reference identifier is an identifier for referring to the association between a plurality of images. The reference identifier is, for example, the file name of the image file 24 of the second image of the reference destination as the reference identifier possessed by the first image when the first image and the second image associated therewith are present. Further, it is the file name of the image file 24 of the first image of the reference source as the reference identifier possessed by the second image. The reference identifier may have an identifier of a group to which the first image and the second image belong. The setting character string is a character string that can be arbitrarily set by the user based on a user operation. The user can set, for example, a character string representing a shooting location in the image when editing and managing the image. Although the setting character string is provided in the association management information, the setting character string may be provided in the metadata 22.

The image file 24 includes metadata 22 and image data 23. The header portion of the image file 24 has the metadata 22, and the body portion has the image data 23. The image data 23 is still image data or moving image data, and is coded data for recording. The metadata 22 is data for managing information including various attribute values regarding the image data 23, in other words, attached information and attribute information. The metadata 22 includes information items such as shooting date/time, shooting location, model, resolution, shooting direction, angle of view, angle of view related information, rotation angle, and thumbnail.

The shooting date/time is, for example, year/month/day/hour/minute/second. The shooting location includes position information or a set character string. The position information includes position coordinates {latitude, longitude, altitude} detected by the position detection unit 113. The setting character string is a character string representing the shooting location set by the user. The model is information that uniquely indicates the type of the imaging device 1. The model may include the lens type and the like. The resolution represents the resolution of the image of the image data 23.

The shooting direction is information including an azimuth angle (denoted by θ) and an elevation angle (denoted by φ). The angle of view is information including the horizontal angle of view (Ah) and the vertical angle of view (Av). The angle-of-view related information is information such as the electronic zoom magnification (n), the focal length (f), the size of the imaging surface of the image sensor, and the like. The focal length f is a 35 mm converted focal length. The dimensions are {horizontal size sw, vertical size sh, diagonal size sd}. Note that the metadata 22 may have a format in which only one of the view angle item and the view angle related information item is provided.

In other words, the rotation angle is a rotation about the optical axis of the lens at the time of image pickup, and is an angle representing the rotation of the image at the time of image pickup. For example, the angle and direction are based on the state where the lateral direction of the housing 400 is aligned with the horizontal plane of the imaging device 1 as shown in FIG. In the case of so-called horizontal shooting, the rotation angle is close to 0 degrees corresponding to the reference state. In the case of an image in which the horizontal angle of view is larger than the vertical angle of view as a ratio of the angle of view, the image becomes horizontally long. In so-called vertical shooting, the rotation angle is close to 90 degrees with respect to the reference state.

The thumbnail is a reduced image based on the original data of the image. The thumbnail is configured according to the resolution and type of the image, and is recorded with or without encoding. When the thumbnail is created and held in advance, the thumbnail can be displayed at high speed during playback display. A plurality of types of thumbnails may be created and held with different resolutions. The thumbnail may be omitted or may be managed separately from the metadata 22. The display control unit 12 may create and display a thumbnail each time reproduction display is performed.

The metadata 22 has a predetermined format, for example, a format extended based on Exif (Exchangeable image file format), but is not limited to this. Exif is a format for photographs of digital cameras. In Exif, the model and shooting condition information are embedded in the image data as metadata. Image display and image editing software can refer to this metadata, and the user can edit the metadata information. The metadata of the conventional Exif image file does not include information such as the angle of view. In the first embodiment, as an expanded format, items such as an angle of view and angle-of-view related information are provided as specific information items in the metadata 22.

The metadata 22 is created and managed as a header portion of the image file 24, but may be created and managed as a metadata file separately from the image data file. Further, not only the specific information is managed using the metadata 22, but also similar information in the management information 21 may be managed collectively. In the case of a moving image, various information such as the metadata 22 may be described for each image frame at a predetermined time interval. The time interval may be constant or may be set only when there is a change in the angle of view.

[Management of relationship between multiple images]
FIG. 6 shows association management regarding the relationship between a plurality of images. The control unit 101 manages the relationship illustrated in FIG. 6 in the association management information of the management information 21 illustrated in FIG. 5, for example. For the sake of explanation, the reference side image is the first image, and the reference side image having a relationship with the first image is the second image. In the example of FIG. 6, images G1, G2, G3, G4, etc. are provided as the first image. The first image may be a wide-angle image or a narrow-angle image. The second image may be a wide-angle image or a narrow-angle image.

The image G1 is a narrow-angle image whose type is a narrow angle, and has an azimuth angle θ1, an elevation angle φ1, a horizontal angle of view Ah1, a vertical angle of view Av1, and the like. The images g11, g12, and g13 are included as the second images associated with the image G1. The images g11, g12, and g13 are narrow-angle images whose type is narrow-angle. For example, the image g11 has an azimuth angle θ11, an elevation angle φ11, a horizontal angle of view Ah11, a vertical angle of view Av11, and the like. The image G1 and the images g11, g12, and g13 are associated as a plurality of images having a predetermined relationship. These images are associated as an image group belonging to the same group, for example, the first group.

The image G2 is a wide-angle image of a wide-angle type, and has an azimuth angle θ2, an elevation angle φ2, a horizontal angle of view Ah2, a vertical angle of view Av2, and the like. The images g21, g22, and g23 are included as the second images associated with the image G2. The images g21, g22, and g23 are narrow-angle images. For example, the image g21 has an azimuth angle θ21, an elevation angle φ21, a horizontal angle of view Ah21, a vertical angle of view Av21, and the like. These images belong to the second group.

The image G3 is a narrow-angle image, and has an azimuth angle θ3, an elevation angle φ3, a horizontal angle of view Ah3, a vertical angle of view Av3, and the like. Images g31 and g32 are included as the second images associated with the image G3. The image g31 is a wide-angle image and the image g32 is a narrow-angle image. These images belong to the third group.

The image G4 is a wide-angle image and has an azimuth angle θ4, an elevation angle φ4, a horizontal angle of view Ah4, a vertical angle of view Av4, and the like. Images g41 and g42 are provided as the second images associated with the image G4. The image g41 is a wide-angle image and the image g42 is a narrow-angle image. These images belong to the fourth group.

Note that, for example, when the image g11 is regarded as the first image on the reference side, the image G1 is the second image on the reference side. Further, for example, the image g11 may be the first image on the reference side, and another image may be associated with the image g11 as a second image.

As an example of the association information, the image file name of each second image is used as a reference identifier provided to the image file of the first image. Further, the image file name of the first image is provided as a reference identifier to be given to the image file of each second image. Further, it has a group identifier as a reference identifier possessed by each associated image. By these, it is possible to mutually refer to the related images.

Note that, as one display mode, information indicating the relationship between a plurality of images as shown in FIG. 6 may be displayed on the display screen. At that time, the first image and the second image may be displayed in a tree structure as shown in FIG. 6, or, although not shown, the nodes of the second image are connected by link lines around the nodes of the first image. May be displayed with.

In the first embodiment, a wide angle and a narrow angle regarding the angle of view are distinguished as the type of image. The imaging device 1 distinguishes the angle of view by an absolute value based on the threshold value. A threshold for the angle of view is set. The control unit 101 sets a wide-angle image as the type when the angle of view of the captured image is equal to or larger than the threshold angle of view, and sets a narrow-angle image when the angle of view is less than the threshold angle of view. The type of view angle is not limited to this, and three or more types may be provided.

[Image shooting direction and angle of view]
FIG. 7 simply shows the concept of the relationship between the shooting direction and the angle of view regarding the wide-angle image and the narrow-angle image as a specific example of the plurality of images. FIG. 7 particularly shows the azimuth angle and the horizontal angle of view. In this example, the image 6a, the image 6b, the image 6c, and the image 6d are included. The images 6a, 6b, and 6c are narrow-angle images, and the image 6d is a wide-angle image. These images are images taken by the user using the image pickup apparatus 1 at the same place, that is, a shooting point P0. For example, it is assumed that each image is captured at a date and time close to each other in the order of image 6d, image 6a, image 6b, and image 6c. As for the shooting date and time of each image, for example, the image 6d is 13:10:13 on January 5, 2016, the image 6a is 13:11:30, the image 6b is 13:14:40 on the same day, and the image 6c is It is 13:15:50 on the same day.

The image 6a has a shooting range 701 corresponding to the shooting direction 7a indicated by the azimuth angle θa and the horizontal field angle Aha. The image 6b has a shooting direction 7b indicated by the azimuth angle θb and a shooting range 702 corresponding to the horizontal field angle Ahb. The image 6c has a shooting direction 7c indicated by the azimuth angle θc and a shooting range 703 corresponding to the horizontal field angle Ahc. The image 6d has the same shooting direction as the shooting direction 7a of the image 6a, and has a shooting range 704 corresponding to the horizontal field angle Ahd. In FIG. 7, of the shooting directions of the image, the shooting direction defined by the azimuth angle θ is indicated by a dashed line.

The point P0 particularly indicates the center point of the lens 402 of the imaging unit 103. The spherical surface 600 indicates a spherical surface of the whole sphere, and is a virtual spherical surface that excludes the point at infinity. The spherical surface 600 shows, in particular, the portion of the circumference corresponding to the azimuth angle θ and the XY plane. N is north, E is east, S is south, and W is west. The X direction corresponds to the east direction, the Y direction corresponds to the north direction, and the Z direction corresponds to the zenith direction. The azimuth angle θ is shown with N as a reference of 0°, clockwise as a positive direction, E of +90°, S of +180°, and W of +270° (-90°).

An imaging range 701 indicated by an arc on the spherical surface 600 indicates a portion of the image 6a corresponding to the horizontal angle of view Aha, and corresponds to an area projected on the spherical surface 600 as viewed from the point P0. In this example, a case where mountains 611, the sky, towers 612, buildings 613, and the like are simply shown on the spherical surface 600 as a landscape imaged. The image 6b is an example of photographing the tower 612, and the image 6c is an example of photographing the building 613.

Specific examples of the values of the azimuth angle θ, the elevation angle φ, the horizontal angle of view Ah, and the vertical angle of view Av of each image are as follows. The image 6a has an azimuth angle θa, an elevation angle φa, a horizontal angle of view Aha, and a vertical angle of view Ava. The image 6b has an azimuth angle θb, an elevation angle φb, a horizontal angle of view Ahb, and a vertical angle of view Avb. The image 6c has an azimuth angle θc and an elevation angle. φc, horizontal angle of view Ahc, and vertical angle of view Avc. Further, the azimuth angle θd=θa, the elevation angle φd=φa, the horizontal angle of view Ahd, and the vertical angle of view Avd of the image 6d.

θa=+10°, φa=0°, Aha=21°, Ava=14°
θb=+45°, φb=0°, Ahb=21°, Avb=14°
θc=−20°, φc=−10°, Ahc=21°, Avc=14°
θd=+10°, φd=0°, Ahd=120°, Avd=80°
Similar to FIG. 7, FIG. 8 is an explanatory diagram particularly showing the elevation angle φ and the vertical angle of view Av. FIG. 8 shows an example of a narrow-angle image which is another image 6e. In FIG. 8, the X direction corresponds to the horizontal direction and the Z direction corresponds to the vertical direction. The positive direction in the Z direction corresponds to the head and zenith directions, and the negative direction corresponds to the foot direction. The elevation angle φ is 0° with reference to the horizontal direction, +90° in the zenith direction of the Z direction, and −90° in the direction of the feet. The image 6e has an azimuth angle θe, an elevation angle φe, a horizontal angle of view Ave, and a vertical angle of view Ahe. A shooting range 705 corresponding to the vertical angle of view Ave of the image 6e is shown. The shooting direction 7e of the image 6e indicates a portion corresponding to the elevation angle φe.

[View angle and view angle related information]
FIG. 9 simply shows the view angle and the view angle related information. Further, FIG. 9 shows a case where the angle of view and the angle of view related information are stored in the setting unit 14 as the setting information. {Horizontal angle of view Ah, vertical angle of view Av} is shown as the angle of view of the image. A horizontal size Sh corresponding to the horizontal angle of view Ah and a vertical size Sv corresponding to the vertical angle of view Av in the image are also shown. The horizontal size Sh is a size from the right end to the left end in the horizontal direction in the image frame. The vertical size Sv is the size from the upper end to the lower end in the vertical direction within the image frame. In the first embodiment, the horizontal view angle Ah and the vertical view angle Av are used, but the present invention is not limited to this, and at least one view angle including the diagonal view angle Ad may be used.

As the angle of view related information, the focal length f of the lens in consideration of the optical zoom, the electronic zoom magnification n, and the dimensions {horizontal size sw, vertical size sh, diagonal size sd} of the image pickup surface of the image pickup device are used. The model may be used instead of the size. The focal length f is a 35 mm converted focal length.

The view angle can also be calculated and obtained based on the relational expression using the view angle related information. When there is sufficient view angle related information to calculate the view angle, the control unit 101 may calculate and obtain the view angle from the view angle related information. The angle of view can be calculated using, for example, the size of the image sensor, the focal length f, and the electronic zoom magnification n. For example, the angle of view can be calculated from a relational expression uniquely determined as arc tangent (arctan) using information such as the focal length of 35 mm or the size of the image sensor.

The following is the case of calculating the angle of view from the angle of view related information. The imaging control unit 11 refers to the view angle related information regarding the image. The imaging control unit 11 can grasp the view angle related information from the setting information of the setting unit 14 described above, or the information described in the metadata 22 or the management information 21. The angle of view {horizontal angle of view Ah, vertical angle of view Av, diagonal angle of view Ad} can be calculated, for example, from the following formula.

Ah=2×arctan(sw/(2×n×f)) (1)
Av=2×arctan(sh/(2×n×f)) (2)
Ad=2×arctan(sd/(2×n×f)) (3)
For example, given the electronic zoom magnification n, the focal length f, and the dimensions {horizontal size sw, vertical size sh}, the horizontal angle of view Ah1 and the vertical angle of view Av1 are determined. The control unit 101 describes the information about the view angle in the view angle item of the metadata 22 or the management information 21. Further, since the angle of view can be calculated from the angle of view related information, the imaging device 1 may store the angle of view related information as one of the metadata 22 instead of storing the angle of view. .. Further, since the dimensions of the image sensor of the angle-of-view related information can be grasped from the model, the image capturing apparatus 1 can store the model as one of the metadata 22 etc. instead of the dimension of the image sensor. Good. For example, the imaging device 1 holds in advance information indicating the correspondence between the model and the view angle related information. Alternatively, the imaging device 1 can refer to the information indicating the correspondence from an external device by communication. The imaging device 1 can obtain the view angle related information from the model based on the information.

When capturing an image, the image capturing apparatus 1 creates information including a capturing date and time, a capturing direction {azimuth θ, elevation φ}, an angle of view {horizontal angle of view Ah, vertical angle of view Av}, and the like. The imaging control unit 11 obtains the shooting date and time by the internal clock 112. The imaging control unit 11 obtains position information {latitude, longitude, altitude} by the position detection unit 113. The imaging control unit 11 obtains the shooting direction {azimuth θ and elevation φ} by the shooting direction measurement unit 111. The imaging control unit 11 obtains the angle of view {horizontal angle of view Ah, vertical angle of view Av} by the setting information or calculation of the setting unit 14. Then, the imaging apparatus 1 describes such information in the corresponding items of the metadata 22 in FIG. 5, and creates the image file 24 together with the image data 23.

[Display screen-Display mode]
FIG. 10 shows the display modes of the display screen 401 of the display unit 41 and their transitions. The imaging device 1 has at least a first display mode, a second display mode, and a third display mode as display modes that are reproduction display modes. The imaging device 1 first displays in the first display mode, for example, and switches the display mode based on a user operation.

As will be described later, the imaging device 1 associates a plurality of images based on the judgment of the shooting location, the shooting date and time of the image, the shooting direction and the angle of view, etc. after shooting or during playback display. Set a group for multiple images and create association management information. When reproducing and displaying an image, the imaging apparatus 1 associates a plurality of images of the same related group in the display screen 401 as follows based on the metadata 22 (at least the shooting direction information) and the association management information. It is displayed in a closed state.

[First display mode-first screen]
The first display mode is a mode for displaying a list of information on all images, and is a mode for displaying a folder screen. On the first screen of the first display mode, information about a plurality of images in a folder is displayed in a list in the order of, for example, a file name, date and time, etc. In this example, information of the images j1, j2, j3 and the like is displayed as a plurality of images arranged in order from the top. In the row of each image, the icon or thumbnail of the image, the file name, the shooting date and time, the type, and other information are displayed. In this example, the types of the images j1, j2, j3 are narrow angles. As other information, attribute values such as a photographing direction {azimuth θ, elevation φ} and an angle of view {horizontal angle of view Ah, vertical angle of view Av} may be displayed. The user can scroll and display the information that cannot be displayed on the first screen. The first screen is not limited to such a folder screen, and various known methods can be used. For example, a screen or the like in which thumbnails of a plurality of images are arranged in parallel in a two-dimensional manner such as in a predetermined direction or vertically and horizontally is possible.

On the first screen, the user selects and operates the desired first image, for example, image j1. The image j1 corresponds to the image G1 in FIG. 6 and the image 6a in FIG. When there is a second image associated with the selected first image, the imaging device 1 causes a transition to the second screen in the second display mode. It is assumed that the image j1 is associated with the images j2 and j3, for example. The images j2, j3, etc. are assumed to correspond to the images g11, g12 in FIG. 6 and the images 6b, 6c in FIG. It is assumed that the relationship between the images j1 to j3 corresponds to the first group in FIG.

[Second display mode-second screen (1)-thumbnail, first group]
The second display mode is a peculiar mode in which association display is performed on a plurality of images. The display control unit 12 displays the second screen based on the image file 24 and the management information 21. On the second screen in the second display mode, first, a predetermined background area 50 is provided as a whole. The background area 50 is a black area or the like and can be set by the user. The background region 50 has a size equal to or larger than the vertical and horizontal sizes of the display screen 401, and has a sufficient size to include the view angle range of a plurality of images.

The display control unit 12 displays a plurality of related images in an associated state on the background area 50 in the display screen 401. The center point of the display screen 401 and the background area 50 is set to point p0. At a point p0 on the background area 50, a graphic 61 representing the image j1 that is the selected first image is displayed. The center point p1 of the figure 61 is located at the point p0. In the first embodiment, the graphic 61 of the first image is a thumbnail. On the background area 50, around the figure 61 of the image j1, a figure 62 showing the image j2 and a figure 63 showing the image j3 are displayed as figures showing the related second image. The center point p2 of the figure 62 and the center point p3 of the figure 63 are shown. The figures 62 and 63 of the second image are, for example, thumbnails. The display position of the graphic 62 or the like of the second image is determined according to the relationship between the display direction of the graphic 61 of the first image and the shooting direction and the angle of view. Also, the display size of each thumbnail is determined to be a size corresponding to the angle of view.

On the second screen, the display state can be changed by operations such as scrolling and enlargement/reduction according to the user operation. For example, by performing an operation such as swipe on the second screen, scrolling and enlargement/reduction in the range corresponding to the background area 50 can be performed. When the display state is changed, the positional relationship between the plurality of images is maintained.

The user selects and operates the desired graphic of the first image or the second image on the second screen. The selection operation is, for example, a tap of a figure. In that case, the imaging device 1 makes a transition to the third screen in the third display mode in which the selected image is displayed. In this example, the case where the graphic 62 of the image j2 which is the second image is selected is shown. In addition, when a predetermined operation for returning to the first display mode is performed on the second screen, the imaging device 1 returns to the first screen in the first display mode. The predetermined operation is, for example, a first display mode designation input, a return button press, a background area 50 tap, or the like.

As described above, on the second screen, a plurality of related images are displayed in the state of thumbnails so that the mutual positional relationship, the relationship including the shooting direction, the angle of view, and the like can be seen at a glance. The user can intuitively understand the relationship between the plurality of images and can easily refer to the second image from the first image. Note that the center points, center lines, and the like shown in the illustration are not displayed, but they may be displayed on the screen as browsing auxiliary information.

[Third display mode-Third screen]
The third display mode is a mode in which detailed display is performed on a narrow-angle image or a wide-angle image that is a designated single image. In this example, an example in which the image j2, which is a narrow-angle image, is enlarged and displayed on the entire third screen is shown. Display of a single wide-angle image is possible as well. First, the center point p2 corresponding to the shooting direction (θ, φ) of the image j2 that is the target image is displayed at the center point p0 of the display screen 401. When the entire image cannot be displayed on the display screen 401, the display control unit 12 may display the image in a part of the image, that is, a part of the angle of view, or may display the image in a reduced size so that the entire image fits. It may be displayed. On the third screen, the display state can be changed by scrolling, enlarging/reducing, or the like according to the user operation. In the case of displaying a wide-angle image, if the whole wide-angle image cannot be displayed, a part of the wide-angle image is first displayed with the shooting direction of the wide-angle image aligned with the point p0, and then displayed according to scrolling or the like. The state is changed. Similarly, when the graphic 61 of the image j1 which is the first image is selected on the second screen, the imaging device 1 displays only the image j1 in detail on the third screen.

When a predetermined operation is performed on the third screen, for example, a second display mode designation input, a return button is pressed, or an image area is tapped, the imaging device 1 returns to the second screen in the second display mode. The transition from the second display mode to the third display mode is not limited to this. An operation mode in which the first display mode is directly changed to the third display mode by a predetermined operation without passing through the second display mode is also possible, and the operation mode can be selected according to the user setting or the like.

[Second display mode-second screen (2)-mark]
FIG. 11 shows another display example of the second screen in the second display mode as a modification. On this second screen, the graphic 61 of the first image is also a thumbnail, and the graphic representing the second image is a predetermined mark. The mark indicates the case of a balloon shape in particular. The mark is variable based on user settings. For example, a mark 72 representing the image j2 is displayed at the center point p2, and a mark 73 representing the image j3 is displayed at the center point p3. When the user selects and operates the mark, the image corresponding to the mark is displayed on the third screen in the third display mode. Alternatively, a thumbnail or the like may be displayed at that position in response to the mark selection operation. The display position of each mark is determined according to the shooting direction (azimuth θ, elevation φ) of the image. In such a form in which the mark is displayed, the screen display content becomes simple.

[Second display mode-second screen (3)-transparent frame]
As a modification, FIG. 12 shows another display example of the second screen in the second display mode. On the second screen, the graphic 61 of the first image is also a thumbnail, and the graphic representing the second image is a transparent frame. In the transparent frame, the four sides of the rectangle are displayed with lines and colors, the inside of which is a transparent region, and the background region 50 is displayed as it is. For example, a transparent frame 82 representing the image j2 is displayed at the center point p2, and a transparent frame 83 representing the image j3 is displayed at the center point p3. When the user selects the transparent frame, the image corresponding to the transparent frame is displayed on the third screen in the third display mode. Alternatively, a thumbnail or the like may be displayed at that position in response to the selection operation of the transparent frame. The display position of the transparent frame is determined according to the shooting direction of the image, and the display size of the transparent frame is determined according to the angle of view of the image.

When the enlargement/reduction is performed on the second screen, the display control unit 12 updates the display size of the thumbnail and the transparent frame so as to correspond to the changed display state. In this way, in the form of displaying thumbnails and transparent frames, it is easy to understand the relationship between the angles of view in a plurality of images.

Further, in FIG. 12, an example regarding the rotation angle of the second image is shown by a transparent frame 84 representing the narrow-angle image at the point p4. This narrow-angle image has an angle α with respect to the x direction as a rotation angle.

Further, the display control unit 12 causes the second image to come to the center of the display screen 401 as a new first image in response to a predetermined operation on the figure of the second image, such as a double tap, in the second screen. The display state may be switched as described above, or a new second image may be referred to and displayed together with it.

[Second display mode-second screen (4)-second group]
FIG. 13 shows another example of a plurality of images as another display example of the second screen. In this example, the image j5 that is the first image selected on the first screen is the wide-angle image. The second image associated with the first image includes images j6 and j7, which are narrow-angle images. The relationship between the plurality of images in this example corresponds to the second group in FIG.

A thumbnail that is a graphic 65 representing the image j5 is displayed such that the center point p5 corresponding to the shooting direction of the image j5 is arranged at the center point p0 on the background area 50. The display size of the thumbnail of the graphic 65 is determined according to the angle of view of the wide-angle image. In this example, the case where the entire thumbnails of the graphic 65 fit within the display screen 401 and the background area 50 is shown. When the entire thumbnail of the graphic 65 does not fit, a part of the thumbnail corresponding to the partial view angle of the wide-angle image is displayed. The entire thumbnail of the graphic 65 can be displayed according to scrolling or the like.

On the figure 65, a transparent frame 86, which is a figure representing the image j6, is displayed so as to overlap with the center point p6, and a transparent frame 87, which is a figure that represents the image j7, is displayed so as to overlap with the center point p7. There is. As described above, even when the first image has a wide angle and the second image has a narrow angle, the relationship between the plurality of images can be grasped on the second screen.

[Second display mode-second screen (5)-third group]
FIG. 14 shows another image as another display example of the second screen. In this example, the image j1 that is the first image selected on the first screen is the narrow-angle image. The second image associated with the first image includes images j8 and j9, and the image j8 is a wide-angle image. The relationship between the plurality of images in this example corresponds to the third group in FIG.

At the center point p0 on the background area 50, a thumbnail that is a graphic 61 representing the image j1 is displayed. On the center point p8 on the background area 50, a transparent frame 88 that is a graphic representing the image j8 is displayed. A transparent frame 89, which is a graphic representing the image j9, is displayed at the center point p9. In this example, the transparent frame 88 of the image j8 extends outside the thumbnail of the image j1. The transparent frame 88 of the image j8 does not fit within the display screen 401, and a part thereof is displayed on the background area 50. According to scrolling or the like, another part of the transparent frame 88 can be displayed. As described above, even when the first image has a narrow angle and the second image has a wide angle, the relationship between the plurality of images can be grasped on the second screen.

[Second display mode-second screen (6)-fourth group]
FIG. 15 shows another image as another display example of the second screen. In this example, the image j5 that is the first image selected on the first screen is the wide-angle image. The second image associated with the first image includes images j8 and j9, and the image j8 is a wide-angle image. The relationship between the plurality of images in this example corresponds to the fourth group in FIG.

At the center point p0 on the background area 50, a thumbnail that is a graphic 65 representing the image j5 is displayed. On the center point p8 on the background area 50, a transparent frame 88 that is a graphic representing the image j8 is displayed. In this example, the thumbnail of the image j5 and the transparent frame 88 of the image j8 partially overlap. The transparent frame 88 of the image j8 does not fit within the display screen 401, and a part thereof is displayed on the background area 50. Thus, even when both the first image and the second image have a wide angle, the relationship between the plurality of images can be grasped on the second screen.

[Second display mode-second screen (7)-scroll]
FIG. 16 shows a scroll example as another display example of the second screen in the second display mode. On the second screen, the background area 50 has a width in the x direction corresponding to a range of 360° as the azimuth angle θ and the horizontal field angle Ah. In the display screen 401, a portion corresponding to a part of the view angle range of the background area 50 is displayed, and the display state can be changed by scrolling in the x direction or the like. A plurality of images are arranged on the background area 50 at positions corresponding to the shooting direction and the angle of view. Initially, the graphic 61 of the first image is arranged and displayed at the point p0. The right end of the background area 50 in FIG. 16 corresponds to θ=180° and the left end corresponds to −180°, and the left and right ends may be connected so that the display loops. The display screen 401b shows an example after scrolling in the x direction. As a result, on the display screen 401b, the graphic of the image included in a part of the background area 50, for example, the graphic 161 is displayed.

[Second display mode-second screen (8)-enlargement/reduction]
FIG. 17 shows an enlargement/reduction example as another display example of the second screen in the second display mode. The upper side of FIG. 17 shows the state before enlargement, and the lower side shows the state after enlargement. On the second screen on the upper side, at the center point p0 on the background area 50, a thumbnail is displayed as the graphic 1701 of the image r1 which is the first image. Examples of the second image associated with the first image include images r2, r3, and r4. The image r2 is a wide-angle image having the same shooting direction as the image r1, and a transparent frame is arranged at the point p0 as a figure 1702. The image r3 is a narrow-angle image whose shooting direction is close to that of the image r1, and the graphic 1703 is arranged at the center point pr3. Since the portion where the image r3 overlaps the thumbnail of the first image becomes large, the graphic 1703 of the image r3 is used as a transparent frame. A transparent frame is arranged at the center point pr4 as a graphic 1704 of the image r4. The display size of each figure is determined based on the angle of view and the enlargement/reduction ratio.

From the above state, a desired portion is enlarged and displayed according to the user's operation such as a pinch. The portion to be enlarged is shown by a frame 1700. In the lower second screen, the frame 1700 is displayed on the entire display screen 401. Each image is enlarged and displayed while maintaining the relationship such as the angle of view. The figure 1701 of the image r1 is displayed as an enlarged image 1701b instead of a thumbnail by enlargement. The enlarged image 1701b is generated based on the original image data 23 or the thumbnail of the image r1, and in the case of high definition display, is generated using the original image data 23.

[Second display mode-second screen (9)-spherical background]
FIG. 18 shows another display example of the second screen in the second display mode of the first embodiment. In the second screen, a circular area corresponding to the spherical surface of a hemisphere is provided as the background area 50 in the display screen 401, instead of being rectangular. A plurality of related images are displayed in the circular background area 50 of the second screen. Images J1 to J5 are provided as examples of a plurality of related images. A circular center point of the background area 50 is designated as a point O. At the point O, a thumbnail is initially displayed as a graphic 601 representing the first image, for example, the image J1. Point O is a point corresponding to the shooting direction (azimuth θ, elevation φ) of the first image. A point at an arbitrary position on the spherical surface of the hemisphere of the background region 50 can be represented by (azimuth angle θ, elevation angle φ). In this example, the case where the point O is (θ=0°, φ=0°) is shown. Further, although orientation information for viewing assistance is displayed at the upper end, lower end, left end, and right end of the circle of the background area 50, it may be omitted. In this example, as the azimuth information, the azimuth angle that changes according to the viewpoint direction within the spherical surface is displayed. The uppermost point is (0°, 90°), the lowermost point is (0°, −90°), the rightmost point is (90°, 0°), and the leftmost point is (0°, −90°). Correspond.

The imaging apparatus 1 determines, for each image in a plurality of related images, a projection image area projected on the spherical surface of the background area 50 in accordance with the viewpoint direction from the shooting position based on information such as the shooting direction and the angle of view. Generated as a graphic representing an image. In this example, images J2 to J5 are provided as the second images associated with the image J1, and figures 602 to 605 representing the respective second images are shown. The figures 602 to 605 are, for example, thumbnails, but may be the transparent frames or marks described above.

The user can view the spherical surface of the celestial sphere from an arbitrary viewpoint direction according to the operation on the second screen. The imaging device 1 changes the display state of a plurality of images in the hemisphere of the background area 50 in real time by known image processing according to the operation of changing the viewpoint. The operation for changing the viewpoint may be scrolling by swiping in the background area 50 or the like, or operation of scroll buttons up and down and left and right outside the background area 50 or similar hardware button operation. Further, the image of the background area 50 can be enlarged or reduced in accordance with an operation such as a pinch. On the second screen, the user can view the relationship between a plurality of images in a state in which the spherical surface of the celestial sphere in the background area 50 is viewed in the desired line-of-sight direction.

At the upper left of the display screen 401, information indicating that the current display mode is the second display mode and that the central first image is the image J1 is displayed. At the lower left of the display screen 401, an item for switching the display of the second image which is the related image is provided. When the item is on, the first image and the second image are displayed as shown in the figure, and when the item is off, the second image is hidden and only the first image is displayed. At the bottom right of the display screen 401, a return button and a display mode switching button are provided. The return button can be used to return to the original state of the first screen, and the display mode switching button can be used to make a transition by selecting a display mode. Also on this second screen, it is possible to display details on the third screen in the third display mode in response to a single image selection operation, and to return to the original state of the second screen in response to a predetermined operation on the third screen. be able to.

Specific examples of the shooting direction and the angle of view of the images J1 to J5 are as follows. The image J1 has (θ, φ)=(0°, 0°) and {horizontal angle of view Ah, vertical angle of view Av}={74°, 53°}. Similarly, the image J2 is (0°, 40°), {14°, 7°}. The image J3 is (45°, 0°), {54°, 38°}. The image J4 is (340°, −30°), {27°, 40°}. The image J5 is (310°, 20°), {17°, 11°}. The image J1 is an image photographed in the north direction of FIG. 7 and the horizontal direction of FIG. 8, and is an image that is horizontally photographed (rotation angle=0°) at a focal length f corresponding to 24 mm in terms of 35 mm. The image J4 is an image that is vertically shot (rotation angle=90°).

The display size of the thumbnail or the transparent frame of each image displayed on the background area 50 is determined based on the angle-of-view information of the image and a predetermined relational expression such as a size proportional to the angle of view. However, as the focal length increases and the angle of view decreases, the thumbnail size also decreases. Therefore, the display size of thumbnails or the like may be set to a minimum value in advance, and the display size may be determined to be equal to or larger than the minimum value.

The figure of the projection image displayed on the hemisphere of the background area 50 is not exactly a rectangle but a figure having distortion according to the radial direction, that is, a figure having a circular arc in the circumferential direction. In this example, the figure of the projected image is simply illustrated as a rectangle. The figure of the projected image may be displayed in a rectangular shape as in this example, or may be displayed in a figure having accurate distortion. The processing method for displaying a distorted graphic is as follows. The display control unit 12 reads the thumbnail in the image data 23 or the metadata 22 of the target image into the memory 133 when generating the graphic to be displayed on the hemisphere of the background area 50. The display control unit 12 uses the signal processing unit 132 to generate a projection image on a hemisphere based on the image data or the thumbnail by using known image processing. The display controller 12 displays a transparent frame corresponding to the projected image or the outer frame line thereof. The processing method when the figure of the projected image is simply displayed as a rectangle is as follows. The display control unit 12 omits the above-described projection image generation processing, and determines the position coordinates of the center point for displaying the graphic of the image on the hemisphere of the background region 50 based on the image capturing direction (θ, φ). To calculate. The display control unit 12 attaches and displays a thumbnail or a transparent frame of the image at the calculated center point.

FIG. 19 shows a display state when the viewpoint direction is the zenith direction, as another display example of the second screen having the circular background area 50. Images K1, K2, and K3 are examples of a plurality of related images. For example, the image K1 has an azimuth angle θ of 0°, the image K2 has θ=90°, and the image K3 has θ=225°. On the second screen, first, the thumbnail of the selected first image is arranged and displayed at a position corresponding to, for example, north facing θ=0°. When the image K2 is selected as the first image, the azimuth θ is −90 while maintaining the relationship of the angle of view of the plurality of images so that the upper end of the circular background region 50 is θ=90°. Displayed in a rotated state. In this display state, it is easy to recognize the relationship between the azimuth angles θ of the plurality of images.

The configurations of the plurality of display examples described above may be appropriately combined or may be configured to transit between the plurality of display examples. For example, the display example of the scroll of the second screen in the second display mode of FIG. 16 and the display example of the enlargement/reduction of FIG. 17 can be applied to an arbitrary screen of the second screen and the third screen. In addition, by performing a reduction display operation on the second screen, the display in the narrow-angle range is gradually changed to the display state in the wide-angle range, and the spherical background of the second screen in FIG. 19 is continuously displayed. You may make it transit to a state. On the contrary, the display state of the spherical background in FIG. 19 may be gradually changed to the display state of the wide angle range in accordance with the enlarged display operation.

Further, on the second screen of FIG. 16, an image not displayed on the screen can be displayed on the screen by scrolling, but instead of this, the following may be performed. The shooting direction measurement unit 111 and the like are operated even in the display mode to detect the orientation of the imaging device 1 or the display device 2. As a result, the image is displayed in such a manner that the orientation of the device and the shooting direction of the image coincide with each other according to the orientation of the device. In other words, the scroll is automatically performed so as to match the orientation of the device.

[Fourth display mode-fourth screen]
FIG. 20 shows a fourth screen in the fourth display mode as a modified example. The fourth display mode is a modification of the first display mode. For example, when the fourth display mode is designated by a predetermined operation such as a display mode switching button on the second screen of FIG. 18 or the like, the fourth screen is displayed. On the fourth screen, thumbnails of a plurality of related images, for example, images J1 to J5 are displayed in a list in a vertical and horizontal arrangement. For example, the thumbnail of the image J1 is displayed on the upper left, and the thumbnail of the image J2 is displayed on the right of the thumbnail. On the first screen, the information of all images is displayed as a list regardless of the relationship between the images, but on the fourth screen, only a plurality of images having the same group relationship are displayed as a list. It is possible to transition from the 4th screen to the 1st screen, the 2nd screen, the 3rd screen, etc. in accordance with a user operation.

When the wide-angle image and the narrow-angle image are mixed on the first screen and the fourth screen, the thumbnails of the wide-angle image and the thumbnails of the narrow-angle image may be vertically and horizontally arranged and displayed. At this time, the display size of the thumbnails may be different display sizes determined according to the size of the angle of view of the image. For example, in the case of a panoramic image in which the horizontal angle of view of the wide-angle image is 180° or more, the thumbnail of the wide-angle image is displayed in one line, and the thumbnails of the plurality of narrow-angle images are displayed in parallel in the line below it. May be. The number of images arranged in parallel is variable in one direction, such as 2, 3, or 4.

[Map display mode]
FIG. 21 shows, as a modification, a display screen when a map display mode is provided as one of the display modes. The image pickup apparatus 1 acquires map information from the outside through the communication network 4 or uses map information stored in advance inside. The imaging device 1 displays map information in the background area 50 in the display screen 401 as a screen corresponding to the map display mode. The imaging device 1 displays a graphic representing each target image on the map of the background area 50 based on the position information of the metadata 22 of the first image and the second image which are the target images. For example, at the center point p0, a thumbnail is displayed as a figure 611 representing the image m1 that is the first image at the shooting point. The graphic 611 may be a transparent frame or a mark. Also, at each point near the point p0, a graphic representing the second image captured at that position is displayed as, for example, a mark. For example, a mark 612 representing the image m2 which is the second image is displayed at the point pm2. These multiple images are associated images that are within a predetermined distance range. On this screen, it is possible to switch to a screen of another display mode according to a predetermined operation of the user. Further, when a graphic at a desired position is selected and operated on this screen, detailed display may be performed on the third screen in the third display mode, or a thumbnail may be displayed at that position.

[Association method]
As the association method, there are a first method and a second method with respect to the timing of performing the association. The first method is a method of associating at the time of image capturing, and the second method is a method of associating at the time of reproducing and displaying. In the first embodiment, the imaging device 1 uses the first method and the display device 2 uses the second method.

In the first method, the imaging device 1 performs the association process using the association unit 15 immediately after capturing an image, and as a result, the association information is described in the metadata 22 or the management information 21. When reproducing and displaying the image, the imaging apparatus 1 can grasp the relationship between the plurality of images by referring to the association information of the metadata 22 or the management information 21, and display the image on the display screen in the second display mode.

In the second method, when the display device 2 reproduces and displays the image file 24 on the display screen, the display device 2 performs the association process by using the association unit 15, and as a result, displays the display screen in the second display mode. In the second method, the display device 2 may omit the description of the association information in the metadata 22 and the management information 21 by performing the association process each time reproduction and display are performed. Alternatively, in the second method, the display device 2 performs the association process at the first time of reproduction and display of an image, describes the association information in the metadata 22 and the management information 21, and then reproduces the image from the second time onward. At the time of display, the association process may be omitted by referring to the association information.

The second method may be adopted in the image pickup apparatus 1 according to another embodiment. Which association method is adopted may be fixed in terms of mounting, or may be a mode in which the association method can be selected and set according to user settings.

[Processing flow (1)]
22 and 23 show processing flows of the image pickup apparatus 1 and the display apparatus 2 according to the first embodiment. FIG. 22 shows a flow of control processing at the time of capturing an image in the image capturing apparatus 1. FIG. 22 has steps S101 to S106. The same processing is repeated each time an image is captured. The steps will be described below in order.

(S101) The control unit 101 sets an angle of view and the like in the setting unit 14 based on a user operation. The imaging control unit 11 grasps the angle of view and the like based on the setting information of the setting unit 14.

(S102) Based on the user operation, the imaging control unit 11 controls the imaging unit 103 to capture an image, obtain the image data 23, and store it in the storage unit 102. The imaging control unit 11 describes the view angle or view angle related information in the metadata 22 of the image file 24 or the management information 21.

(S103) Branching is performed according to the first method for associating at the time of image capturing or the second method for associating at the time of playback display. In the case of the first method, the process proceeds to S104, and in the case of the second method, the process proceeds to S106. That is, since the image pickup apparatus 1 of the first embodiment adopts the first method, the process proceeds to S104, and when the image pickup apparatus 1 of the other embodiment adopts the second method, the process proceeds to S106.

(S104) The associating unit 15 performs a predetermined associating process. Thereby, a plurality of images including the first image and the second image are associated with each other.

(S105) The associating unit 15 creates association information as a result of the association process and describes it in the metadata 22 of the image file 24 or the management information 21. The imaging control unit 11 stores the image file 24 in the storage unit 102 and records it in the recording medium 107 of the recording unit 106.

(S106) Since the imaging control unit 11 does not perform association at this point, it creates the metadata 22 and the management information 21 that do not include association information. The control unit 101 stores the image file 24 in the storage unit 102 and records it in the recording medium 107 of the recording unit 106.

[Processing flow (2)]
FIG. 23 shows a flow of control processing at the time of reproducing and displaying an image in the image pickup apparatus 1 or the display apparatus 2. The control of FIG. 23 corresponds to the control of changing the display mode as shown in FIG. FIG. 23 has steps S201 to S209. The steps will be described below in order. Hereinafter, the case where the imaging device 1 is the main body will be described, but the same applies when the display device 2 is the main body.

(S201) Based on a user operation, the control unit 101 causes the display control unit 12 to display the first screen in the first display mode on the display screen 401 of the display unit 41 of the display device 104 as shown in FIG. The display control unit 12 receives selection of an image in the first screen and the like according to a user input operation. The selected image is the first image on the reference side.

(S202) The display control unit 12 branches depending on whether it is the first method or the second method. If it is the second method, proceed to S203, and if it is the first method, proceed to S205. That is, in the case of the image pickup apparatus 1 according to the first embodiment, the process proceeds to S205 because it is the first system, and in the case of the display device 2, the process proceeds to S203 because it is the second system.

(S203) The display control unit 12 uses the associating unit 15 to perform a process of associating a plurality of images immediately before the reproduction display.

(S204) If the result of S203 is that there is a second image associated with the first image, the display control unit 12 reads the image data 23 of that second image.

(S205) In the case of the first method, association processing has already been performed. The display control unit 12 confirms the association by referring to the association information in the metadata 22 of the image file 24 of the first image or the management information 21. The display control unit 12 reads out the image data 23 of the second image associated with the first image.

(S206) The display control unit 12 displays a plurality of images including the first image and the second image in the second display mode on the display screen using the image data 23 obtained as a result of S204 or S205. That is, the display control unit 12 displays the graphic of the first image at the center position on the background area 50 and the graphic of the second image around it, as in the example of the second screen in FIG. 10. The display control unit 12 accepts selection of a figure of the first image or the second image in the second screen in response to a user input operation. For example, when only the first image is browsed or when only the second image is browsed, the corresponding graphic selection operation is performed. When returning to the first screen, a predetermined operation is performed.

(S207) The display control unit 12 branches depending on whether the graphic of the first image or the second image is selected on the second screen in the second display mode or an operation such as returning is performed. When the first image or the second image is selected, the process proceeds to S208. If the operation is another operation, such as pressing the return button, specifying the first display mode, or tapping the background area 50, the process returns to S201.

(S208) The display control unit 12 uses the image data 23 of the image corresponding to the graphic selected in S207 to display the image in detail on the third screen in the third display mode. The display control unit 12 receives a user input operation on the third screen. For example, when returning to the second screen, the return button is pressed, the second display mode is designated, the image area is tapped, and the like.

(S209) The display control unit 12 returns to S206 as shown by A1 when there is an operation such as returning on the third screen in the third display mode, and when there is no operation, in S208, the third display mode is set. Continue the display. In addition, you may make it return to a 1st screen of a 1st display mode from a 3rd display mode according to a predetermined operation.

[Association processing (1)]
The associating unit 15 performs a process of associating a plurality of images. The association process includes a first process and a second process. The first process is a process of selecting an image that is a candidate for association based on a selection condition of a predetermined selection method. As the predetermined selection method, there are the following first selection method, second selection method, third selection method, and the like. In the first embodiment, the first method is used. In the modification, the second selection method or the third selection method is used. It should be noted that which selection method is adopted may be fixed in terms of mounting or may be variably selected by user setting.

[First selection method-shooting location]
FIG. 24 is an explanatory diagram showing the first selection method. The first selection method uses a condition regarding a shooting location as a selection condition. In the first selection method, images whose shooting locations are determined to be substantially the same are selected as candidates. In the first selection method, position information {latitude, longitude, altitude} by the position detection unit 113 is used to determine the shooting location. When associating, the associating unit 15 refers to the position information {latitude, longitude} described in the shooting location item of the metadata 22 at the time of shooting, for example. {Latitude, longitude} is represented by {LA, LO}.

As shown in FIG. 24, the associating unit 15 takes the position 241 representing the shooting location of the first image as the center point in space, for example, the XY plane, with the first image as the reference. It is assumed that the first image is image k1 and each of the second images is images k2 to k4. The position 241 is represented by {LA1, LO1}. The position range 240 corresponds to the threshold value of the condition, and indicates a circular region centered on the position 241 and having a predetermined radial distance. The radial distance defining the position range 240 may be a fixed set value in design or a set value that can be changed by user setting. The position range 240 is not limited to a circle and is possible.

The associating unit 15 compares the position 241 of the first image with the position of each image that is the second image. In this example, there is shown a case where there is a position 242 {LA2, LO2} of the image k2, a position 243 {LA3, LO3} of the image k3, and a position 244 {LA4, LO4} of the image k4 around the first image. The associating unit 15 searches for an image included in the position range 240 with respect to the position 241 of the first image. In this example, the position range 240 includes the position 242 of the image k2 and the position 243 of the image k3, but does not include the position 244 of the image k4. The associating unit 15 determines that the second image whose position is included in the position range 240 is an image captured at the same shooting location as the first image, and selects it as a candidate for association. An example of the same shooting location is a case where the user shoots in each direction of 360° while circling the circumference along a handrail on a certain observation deck.

The method for determining the same shooting location is not limited to the method using GPS position information, but a method using position information of a short-range communication interface such as BlueTooth (registered trademark) or position information such as an indoor positioning system is used. The method etc. are also applicable. For example, a device corresponding to the short-range communication interface is installed in a facility or the like, and the imaging device 1 acquires position information by communicating with the device. As a result, it is possible to grasp the position with high accuracy.

[Second selection method-shooting date and time]
FIG. 25 is an explanatory diagram showing the second selection method. The second selection method uses a condition regarding the shooting date and time as the selection condition. In the second selection method, images whose shooting dates and times are judged to be roughly the same are selected as candidates. When associating, the associating unit 15 refers to, for example, the shooting date/time (year/month/day/hour/minute/second) described in the shooting date/time item of the metadata 22.

The associating unit 15 takes a time point t1 representing the shooting date and time of the first image with the first image as a reference in the time series of the shooting date and time. In this example, the time point t1 is the date and time DT3. The date/time range 250 corresponds to the threshold value of the condition, and indicates a range based on a predetermined time, including the time point t1 as the center point and before and after that time point. The time defining the date/time range 250 may be a fixed set value in design or a set value that can be changed by user setting. The date/time range 250 can be set to, for example, one day (24 hours), one hour, five minutes, or the like.

The associating unit 15 searches for the second image included in the predetermined date and time range 250 with respect to the time point t1 of the shooting date and time of the first image. The associating unit 15 compares the shooting date and time of each image, which is the second image, with the time t1 of the shooting date and time of the first image. In this example, a case is shown in which there are a time point t2 of the shooting date and time of the image k2, a time point t3 of the image k3, and a time point t4 of the image k4 near the time point t1. In this example, the date and time range 250 includes the time point t2 of the image k2 and the time point t3 of the image k3, but does not include the time point t4 of the image k4. The control unit 101 determines that the second image whose time point is included in the date and time range 250 is an image captured at the same shooting date and time as the first image, and selects it as a candidate for association.

[Third selection method-Setting character string]
FIG. 26 is an explanatory diagram showing the third selection method. The third selection method uses, as a selection condition, a condition regarding a set character string that can be arbitrarily set by the user. In the third selection method, images whose set character strings are judged to be substantially the same are selected as candidates. When associating, the associating unit 15 refers to the character string set as the setting character string of the association management information after imaging, for example.

FIG. 26 shows an example of a list display of information of a plurality of images on the first screen in the first display mode. In this example, for each image, a file name, shooting date, tag, type, etc. are displayed as information. The “tag” item corresponds to the setting character string. The user can arbitrarily set a character string in the "tag" item by editing work. In this example, the user sets a character string representing the shooting location in the “tag” item. For example, the images k1, k2, k3 are images taken in the same park, and the character string “A park” is set. The images k4 and k5 are images taken at the same observatory, and the character string "B observatory" is set. The "type" item displays "wide angle" or "narrow angle" as the type of the angle of view of the image.

The associating unit 15 refers to the first image selected by the user, for example, the image k1, as a reference, and refers to the set character string of the “tag” item of the first image. The associating unit 15 refers to and compares the setting character strings of the first image with the setting character strings of the other images that are the other second images, and, for example, an image having a setting character string that includes the same character string. Look for. For example, the set character strings of the images k1, k2, and k3 are the same. The associating unit 15 determines that these images are images captured at the same shooting location, and selects them as candidates for association.

As a modified example, the control unit 101 may automatically set a character string indicating a shooting location at the time of capturing an image, etc. without depending on a user operation. For example, the control unit 101 may determine the character string representing the shooting location based on the position information {latitude, longitude}. The control unit 101 may set a character string in the “file name” item or the like of the image.

The above selection methods can also be applied in combination. For example, the condition of the shooting location of the first selection method and the condition of the shooting date and time of the second selection method may be combined. As the condition, it is possible to specify, for example, "up to a year ago in the same shooting location".

[Fourth selection method-type of subject]
The following fourth selection method is also possible as another selection method. The image pickup apparatus 1 determines the type of subject by image recognition processing. The types of subjects include landscapes, buildings, people, animals, and the like. The type of subject that can be determined depends on the accuracy of image recognition processing and the like. The imaging apparatus 1 estimates a plurality of images having the same subject type as a group having a relationship and selects them as candidates according to the determined subject type.

Further, the imaging device 1 may use the determined type of subject as a reference for determining the relationship between a plurality of images. The imaging device 1 adjusts, for example, the position range, which is a threshold value of the first selection method, according to the type of subject. For example, when the type of subject is landscape, the influence of image changes is small even if the focal length is large and the difference in the positions of the shooting points is large. Therefore, in this case, the position range is set relatively large. Thereby, the image group included in the large position range is associated. When the type of subject is a person, if the focal length is short and the difference in the position of the shooting point is large, the influence on the image change is large. Therefore, in this case, the position range is set relatively small.

Similarly, the imaging device 1 may adjust the threshold value such as the position range according to the determination of the distance to the subject and the difference in the shooting mode. For example, when the shooting mode is set to the macro mode or the like suitable for landscape shooting, the position range that is the threshold value is relatively increased.

[Association process (2)]
The second process in the associating process of the associating unit 15 is as follows. In the second process, with respect to the image selected as the candidate in the first process, the association determination is performed using the information on the shooting direction and the angle of view, and the association is performed according to the determination result. The associating unit 15 refers to the metadata 22 of each image file 24 or the information on the shooting direction and the angle of view in the management information 21. As a first condition, the associating unit 15 determines that the shooting direction (azimuth θ, elevation φ) of the candidate image is within the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av} of the first image as a reference. Judge whether it is included or not. The associating unit 15 determines an image that satisfies the first condition as an image to be associated. In addition, as a second condition, the associating unit 15 extends the outside of the range of the angle of view {horizontal angle of view Ah, vertical angle of view Av) of the first image, which is a reference, to a candidate within the predetermined angle of view range. It may be determined whether or not the shooting direction (azimuth θ, elevation φ) of the image is included. The associating unit 15 determines an image that satisfies the second condition as an image to be associated. In the second process of association, the first condition or the second condition is applied.

An example of satisfying the first condition is as follows. As an example of the first image, the image 6d that is the wide-angle image in FIG. 7 is used. The horizontal field angle Ahd of the image 6d is 120°, and the shooting range 704 of the horizontal field angle Ahd is 70° at the right end position and −50° at the left end position. The shooting range 704 includes the azimuth θb=45° of the image 6b in the shooting direction 7b and the azimuth θc=−20° of the image 6c in the shooting direction 7c. Although not shown, the elevation angle φb of the image 6b in the shooting direction 7b and the elevation angle φc of the image 6c in the shooting direction 7c are included in the range of the vertical angle of view Avd of the image 6d. As a result, the first condition is satisfied, and the image 6d is associated with the image 6b and the image 6c.

FIG. 27 is an explanatory diagram of the first condition, the second condition, and the like when determining the relationship between a plurality of images. In the entire background area 50 in the display screen 401, the horizontal direction in the screen corresponding to the azimuth angle θ is the x direction, and the vertical direction in the screen corresponding to the elevation angle φ is the y direction. The horizontal width of the display screen 401 in the x direction is W, and the vertical width in the y direction is H. The center point p0 of the display screen 401 is represented by position coordinates (x0, y0). It is assumed that the background area 50 having the same size or more is set on the entire display screen 401.

As the first image that is the reference, the image q1 that is the first narrow-angle image is taken as an example. The point corresponding to the shooting direction (θ, φ) of the image q1 is the center point p1. In this example, (θ, φ) at the center point p1 is (0°, 0°). The figure 271 representing the image q1 is arranged such that the center point p1 is aligned with the point p0. The graphic 271 is rectangular here and is shown by a solid line frame. The horizontal angle of view Ah1 of the image q1 is 21° and the vertical angle of view Av1 is 14°. The figure 271 of the image q1 has a horizontal size Sh1 corresponding to the horizontal angle of view Ah1 and a vertical size Sv1 corresponding to the vertical angle of view Av1. The right end in the x direction of the figure 271 is shown as a position x11 and the left end is shown as a position x12, and the azimuth angles θ are 10.5° and -10.5°, respectively. The upper end of the figure 271 in the y direction is indicated by the position y11 and the lower end thereof is indicated by the position y12, and the elevation angle φ is 7° and −7°, respectively.

As the second image, the image q2 that is the second narrow-angle image is taken as an example. The point corresponding to the shooting direction (θ, φ) of the image q2 is defined as the center point p2. In this example, (θ, φ) of the center point p2 is (16°, −6°). A graphic 272 representing the image q2 is arranged at the center point p2 and is also shown by a solid line frame. The horizontal angle of view Ah2 and the vertical angle of view Av2 of the image q2 are set to 21° and 14° similarly to the image q1. The figure 272 of the image q2 has a horizontal size Sh2 corresponding to the horizontal angle of view Ah2 and a vertical size Sv2 corresponding to the vertical angle of view Av2. The right end of the figure 272 in the x direction is indicated by a position x21 and the left end thereof is indicated by a position x22, and the azimuth angles θ are 26.5° and 5.5°, respectively. The upper end of the figure 272 in the y direction is indicated by the position y21 and the lower end thereof is indicated by the position y22, and the elevation angle φ is 1° and −13°, respectively.

An example of the first condition is that the shooting direction (θ, φ) of the image q2 is included within the range of the horizontal view angle Ah1 and the vertical view angle Av1 centered on the shooting direction of the image q1. In other words, the center point p2 of the image q2 is included within the range of the horizontal size and the vertical size of the image q1. In particular, it is assumed that both the azimuth angle θ and the elevation angle φ need to satisfy the conditions. In the present example, the azimuth angle θ=16° of the image q2 is outside the range (-10.5° to 10.5°) of the horizontal field angle Ah1 of the image q1, and the elevation angle φ=-6° is vertical. It is within the range of the angle of view Av1 (−7° to 7°). Therefore, the image q2 does not satisfy the first condition.

For the second condition, the extended threshold range 273 is set outside the range of the horizontal angle of view Ah1 and the vertical angle of view Av1 centered on the shooting direction of the image q1 which is the first image. The threshold range 273 is indicated by a broken line frame. The threshold value range 273 may be a fixed set value for mounting or a user set value. The threshold value range 273 may be particularly adapted to the size of the background area 50. Further, in particular, the threshold range 273 may be a range obtained by a predetermined calculation formula based on the size of the angle of view of the first image, for example, a range obtained by adding a predetermined angle of view or a range obtained by multiplying by a predetermined rate. Good. In this example, the threshold range 273 is a range three times larger than the range of the angle of view of the first image, and the horizontal angle of view Aht=63° and the vertical angle of view Avt=42°. The right end in the x direction of the threshold range 273 is the position x31, the left end is the position x32, and the azimuth angles θ are 31.5° and -31.5°. The upper end of the threshold range 273 in the y direction is the position y31, the lower end is the position y32, and the elevation angle φ is 21° and -21°.

An example of the second condition is that the shooting direction (θ, φ) of the image q2 is included in the threshold range 273 based on the angle of view of the image q1. In particular, it is assumed that both the azimuth angle θ and the elevation angle φ need to satisfy the conditions. In this example, the azimuth angle θ=16° of the image q2 is within the range of the horizontal field angle Aht of the threshold range 273, and the elevation angle φ=−6° is within the range of the vertical field angle Avt. Therefore, the image q2 satisfies the second condition. Thereby, when the second condition is used, the image q2 is associated with the image q1.

The other conditions are as follows. As a third condition, it is assumed that a part of the view angle range of the second image overlaps the view angle range of the first image. As a fourth condition, it is assumed that a part of the view angle range of the second image overlaps the threshold range outside the view angle range of the first image. For example, part of the view angle range of the image q2 overlaps with the view angle range of the image q1. In other words, the position x22 at the left end of the image q2 is on the left of the position x11 at the right end of the image q1. Therefore, the image q2 is associated with the image q1.

As a fifth condition, a difference value between an angle (θ or φ) in the shooting direction of the first image and a corresponding angle (θ or φ) in the shooting direction of the second image is within a predetermined threshold angle. To do. In other words, the distance between the position coordinates of the center point of the first image and the center point of the second image is calculated, and the distance is within a predetermined threshold distance. For example, the difference value dθ of the azimuth angle θ between the center point p1 of the image q1 and the center point p2 of the image q2 is 16°, and the difference value dφ of the elevation angle φ is −6°. The threshold angle is (dθt, dφt). When dθ≦dθt or dφ≦dφt, the condition is satisfied.

Even when the first image is a wide-angle image or the second image is a wide-angle image, it is possible to determine the association by applying the above-described conditions in the same manner. Further, in particular, the first condition may be applied when the first image is a wide-angle image, and the second condition may be applied when the first image is a narrow-angle image. As described above, a mode in which conditions are selected according to the type of image, or a mode in which a plurality of conditions are combined and applied may be used.

[Video]
Although the case of a still image has been mainly described above, it is basically applicable to a case of a moving image. When shooting a moving image, the imaging control unit 11 obtains image data 23 of the moving image by shooting while changing the electronic zoom magnification, the angle of view, and the like in consecutive shooting modes. The imaging control unit 11 describes, for the image file 24 of the moving image, information such as the electronic zoom magnification and the angle of view according to the time point in the metadata 22 or the management information 21.

When the moving image is reproduced and displayed on the second screen in the second display mode, the display control unit 12 changes the shooting date and time, the position information described in the metadata 22 and the shooting direction according to the time. The display position of the figure is determined based on. The display position may be, for example, one fixed display position according to the shooting direction of the first image frame forming the moving image, or may be a display position determined from statistical values of a plurality of image frames. The display position may be a display position corresponding to the shooting direction that changes according to the image frame.

[Effects, etc.]
As described above, according to the imaging device 1 and the display device 2 of the first embodiment, it is possible to easily understand the relationship between a plurality of images including a wide-angle image and a narrow-angle image, and to realize better usability. On the display screen, for example, a narrow-angle image having a relationship is automatically overlapped and displayed in the vicinity of a wide-angle image, so that the user can easily recognize a plurality of related images and intuitively grasp the images. It can be viewed appropriately. The user can easily understand the relationship between the wide-angle image and the narrow-angle image. The user can quickly refer to the related second image while viewing the first image. The user can suitably browse a large number of images while switching the display mode.

The following are examples of modifications of the first embodiment. As a modified example, the relationship between the angles of view may be determined relative to the type of image. That is, the imaging device 1 or the display device 2 refers to the angle of view in a plurality of images, sets the image having the maximum angle of view as the wide-angle image, and the images other than the wide-angle image as the narrow-angle image. To do.

[Modification (1)-second screen]
FIG. 28 shows an example of the second screen in the second display mode in the image pickup apparatus 1 of the modification of the first embodiment. In the display screen 401, a main region 281 and a region other than the main region 281 are provided, in this example, a right side region 282 and a lower side region 283. In the main area 281, a thumbnail is displayed as a figure 284 of the first image in the center position on the background area, and a figure of the second image associated around the thumbnail is displayed. As the graphics of the first image and the second image, different graphics may be displayed depending on the type of the image, the shooting date and time, and the like. In this example, circles, diamonds, and stars are shown as figures. As the type of image, for example, a type according to the difference in the angle of view may be set based on the user setting. Information may be added to the graphic and displayed. In this example, the mark is given an identification number. A mark with an identification number of 1 is displayed together with the thumbnail of the first image. As the figure of the second image, a mark having an identification number of 2 or less is displayed. In addition, an attribute value based on the metadata 22 or the management information 21 may be displayed as information on the graphic, or only the attribute value may be displayed.

In the area 282 on the right side, information of all the plurality of associated images, such as marks and attribute values, is displayed in parallel. In the area 281, the graphics of some images are displayed and the graphics of other images are not displayed as a display state based on a user operation. The area 282 also displays information about images that are not displayed in the area 281. When a graphic of the image is selected in the area 281 or the area 282, the display mode may be changed to the third display mode. In the lower right area, the thumbnail of the last selected image is displayed as a preview.

In the area 283 on the lower side, display mode information and attribute value information of the first image are displayed. Further, an operation button for switching display and non-display of the mark in the second display mode is provided. The mark is displayed in the on state, and the mark is hidden in the off state. Further, a return button or the like for returning to the first display mode is provided.

Further, in the second screen, when a plurality of associated second images are densely present in the same position in the background area and in the same area, individual figures are displayed in an overlapping manner. Instead, display one representative mark. In this example, the representative mark is a star. When the user selects and operates the representative mark, information on the plurality of second images associated with the position of the representative mark is displayed. At that time, for example, the pop-up area 285 is displayed, and the pop-up area 285 is displayed with the information of the plurality of second images. When the user selects a desired second image in the pop-up area 285, a transition is made to the third display mode in which the selected second image is displayed in detail.

As another method, information of a plurality of second images associated with the representative mark may be displayed in the area 282 on the right side. Further, a plurality of associated second images may be sequentially displayed, that is, a slide show, in a predetermined area in accordance with the selection of the representative mark. For example, when there are a plurality of images taken at the same location and in the same shooting direction and at different shooting dates and times, those images can be associated and continuously reproduced and displayed. In addition, an area in which a plurality of second images are densely displayed may be enlarged and displayed in accordance with a selection operation of the representative mark so that the figures of the plurality of second images can be distinguished.

[Modification (2)-tilt operation]
As a modified example, the tilt angle representing the tilt and rotation of the housing of the image pickup apparatus 1 is detected using a sensor or the like of the shooting direction measurement unit 111. The imaging device 1 controls switching of display modes, scrolling of a plurality of images in the second screen, and the like according to the state of the detected tilt angle. The display mode in which this function is valid and the display mode in which this function is invalid can be switched according to a predetermined operation. Normally, in the invalid display mode, the display content of the second screen is fixed and displayed regardless of the tilt angle of the imaging device 1. Then, the display state can be changed by scrolling or enlarging/reducing in accordance with the above-mentioned predetermined operation.

FIG. 29 shows a state in which the display screen 401 of the housing 400 of the image pickup apparatus 1 is arranged on the XZ plane in the effective display mode. With respect to the directions of the housing 400 and the display screen 401, an operation of tilting the y-axis and its tilt angle 291 and an operation of tilting the x-axis and its tilt angle 292 are shown. As an example of the second screen, an example similar to the second screen of FIG. 10 is shown. On the background area 50 of the second screen, a graphic 61 of the first image is displayed in the center and a graphic 62 of the second image is displayed nearby.

In this state, an operation of tilting the housing 400 is performed as a user operation. For example, an operation of tilting to the right in the positive direction with respect to the y axis is performed. The imaging device 1 detects the tilt angle 291 at that time, and changes the display state in the second screen by scrolling to the right according to the tilt angle 291. As a result, each image is moved to the left in the second screen, and at the same time, the information on the right outside the screen is displayed in the screen. In this display mode, the user can easily change the display state only by tilting, and can view a plurality of images with little effort. Similarly, it is also possible to adopt a form in which the enlargement/reduction is controlled according to the tilt operation.

[Modification (3)-External device cooperation]
The imaging device 1 and the display device 2 may store the image file 24 in an external device, or may read the image file 24 from an external device and reproduce and display it. The imaging device 1 and the display device 2 may perform the association process for the image file 24 to be transmitted to the external device, and may transmit the image file 24 including the association information to the external device. Further, the imaging device 1 and the display device 2 may perform the association process on the image file 24 received from the external device, and may reproduce and display the image file 24 in the associated state based on the association information. That is, as an association method other than the above-mentioned first method and second method, an association method is performed before the image file 24 is transmitted to an external device, or an association method is performed after the image file 24 is received from the external device. May be adopted. When transmitting a plurality of image files 24 to an external device, the imaging device 1 and the display device 2 may describe the angle-of-view information and the association information as the metadata 22 for each image file 24, or the plurality of image files 24 may be described. It may be provided by describing it in a management file different from the image file 24.

For example, in the system of FIG. 1, the imaging apparatus 1 performs an association process on a plurality of captured images before transmitting them to the server 3, creates an image file 24 including an angle of view and association information, and communicates with the communication network 4. To the server 3 for storage in the image data DB. The imaging device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. At this time, a plurality of image files 24 can be collectively obtained by designating the shooting date and time, the shooting location, and the like as search conditions. The imaging device 1 or the display device 2 reproduces and displays the acquired image data 23 of the image files 24 in an associated state based on the reference of the association information.

In addition, for example, the imaging device 1 transmits the image file 24 that includes the angle of view and does not include the association information to the server 3 and saves it in the image data DB without performing the association process on the plurality of captured images. The imaging device 1 or the display device 2 accesses the server 3 through the communication network 4 and acquires the image file 24 from the image data DB of the server 3. The imaging device 1 or the display device 2 performs an association process on the acquired image files 24, and reproduces and displays them in an associated state based on the angle of view and the association information.

Further, the server 3 may perform the association process for the plurality of image data 23 in the image data DB. The server 3 performs the association process and saves it as an image file 24 including association information. The imaging device 1 or the display device 2 reproduces and displays the plurality of image files 24 acquired from the server 3 based on the reference of the association information.

In addition, for example, the imaging device 1 performs an association process on a plurality of captured images and transmits an image file 24 including an angle of view and association information to the display device 2. The display device 2 acquires the image file 24, and reproduces and displays the image data 23 of the acquired image file 24 in an associated state based on the reference of the association information.

Further, for example, the imaging device 1 transmits the image file 24 including the angle of view and not including the association information to the display device 2 without associating the captured images with each other. The display device 2 acquires the image file 24, performs an association process on the image data 23 of the acquired image file 24, and reproduces and displays the image data 23 in an associated state based on the association information.

Further, the imaging device 1 and the display device 2 may read the respective image files 24 from a plurality of devices of the same user and integrate and manage them. For example, the first image capturing apparatus 1A and the second image capturing apparatus 1B in FIG. 1 each transmit the image file 24 of the captured image to the first display apparatus 2A without making an association. The first display device 2A performs the associating process on the image files 24 and reproduces and displays the images in the associated state.

According to this modification, the association display function can be realized in a system in which a plurality of devices cooperate, and the user can realize association and storage of many images.

(Embodiment 2)
The imaging device and the display device according to the second embodiment of the present invention will be described with reference to FIGS. 30 and 31. Hereinafter, parts of the configuration of the second embodiment different from the configuration of the first embodiment will be described. The image pickup apparatus 1 according to the second embodiment is provided with a specific shooting mode for associating a plurality of images.

[Shooting mode]
The imaging device 1 according to the second embodiment has, as shooting modes, a first shooting mode that is a normal shooting mode and a second shooting mode that is a group shooting mode for making association. The imaging device 1 switches the shooting mode based on a user input operation or a user setting. The user performs a predetermined operation to switch between the first shooting mode and the second shooting mode at an arbitrary timing through the operation input unit 105 and the like. The predetermined operation may be, for example, pressing a mode switching button.

FIG. 30 illustrates switching of shooting modes on the time axis in the image pickup apparatus 1 according to the second embodiment. The image pickup apparatus 1 is initially in the state of the first shooting mode, and does not perform the associating process with respect to each image taken in the first shooting mode. For example, the image captured at the time point t11 is not associated, and the association information is not described in the metadata 22 or the like.

The user performs an operation of switching from the first shooting mode to the second shooting mode at a desired timing, for example, time T2. In the state of the second image capturing mode, the image capturing apparatus 1 automatically associates the individual images captured during that period as a group. For example, images are captured at time points t1 to t4 in the second shooting mode. The control unit 101 causes the associating unit 15 to perform the associating process on the images, and describes the associating information in the metadata 22 or the management information 21. In this example, the images at time points t1 to t4 are associated as one group. The group identifier is described in the image file 24 or the management information 21 of those images. The group identifier may be a character string using the start date and time and the end date and time of the period of the second shooting mode.

The user performs an operation of switching from the second shooting mode to the first shooting mode at a desired timing, for example, time T4. The image capturing apparatus 1 does not associate each image captured in the first capturing mode, for example, the image captured at time t12. The control unit 101 may execute the association process when returning to the first shooting mode. At the time of reproduction display, the display control unit 12 confirms the association based on the reference of the image file 24 or the group identifier in the management information 21 and performs the association display.

[Menu screen]
FIG. 31 shows a menu screen relating to the shooting mode as an example of the display screen of the image pickup apparatus 1. This menu screen is displayed, for example, by pressing the center button of the cross key 433 in FIG. This menu screen has a shooting menu 311, a reproduction menu 312, and a setting menu 313. The shooting menu 311 has setting items related to the shooting mode, the reproduction menu 312 has setting items related to the reproduction display mode, and the setting menu 313 has setting items related to the setting mode. It can be set by the user in each setting item of each mode. For example, in the shooting menu 311, setting items include group shooting 314, size, image quality, focus mode, image stabilization, and the like. Size, image quality, etc. are known setting items.

In the menu screen, the user selects a setting item for group shooting 314 in the shooting menu 311 in response to a tap operation or the like, and selects the ON button to set the group shooting mode as the shooting mode. When the off button is selected, the group shooting mode is released. The group shooting mode may be turned on/off by a dedicated hardware button. Further, the imaging apparatus 1 may automatically switch the group shooting mode to the off state when the power is turned off in the group shooting mode on state. In addition, the imaging device 1 may automatically switch the group shooting mode to the off state when a predetermined time has elapsed from the time when the group shooting mode is turned on or the time when the image is shot in the shooting mode. Good.

[Effects, etc.]
As described above, according to the second embodiment, the user can easily switch the shooting modes and associate a plurality of images with each other. The second embodiment corresponds to a mode in which the judgment regarding the shooting date and time of the above-mentioned second selection method is automated by using a unique shooting mode.

(Embodiment 3)
An imaging device and a display device according to the third embodiment of the present invention will be described with reference to FIG. Hereinafter, parts of the configuration of the third embodiment different from those of the first embodiment will be described. The image pickup apparatus 1 according to the third embodiment has a function of displaying captured images in association with each other in the monitor image displayed on the display screen in the shooting mode. This function realizes an assist function that supports the shooting operation including the user's determination of the shooting direction and the like. In particular, in the third embodiment, the group shooting mode of the second embodiment is used together. In the group shooting mode as the shooting mode, while displaying the monitor image, the shot image is displayed in association with the second image.

[Shooting monitor]
32 shows the display screen 404 of the EVF 403 of FIG. 4 in the first monitor method in the third embodiment. The display control unit 12 displays the monitor image in the shooting mode and the associated image on the display screen 404. In the first monitor method, the monitor image is regarded as the background area 50 and the first image, and the graphic of the second image is displayed on the monitor image. The imaging device 1 displays the monitor image 411 corresponding to the shooting direction and the angle of view during shooting in association with the entire background area 50 of the display screen 404. The center point p0 of the display screen 404 and the center point of the monitor image 411 match.

The user sets the group shooting mode and starts shooting. The imaging device 1 is set to the group shooting mode according to a predetermined operation. As in the second embodiment, the image capturing apparatus 1 associates the images captured in the group capturing mode as a group, and holds the associated second image information in the storage unit 102 or the like.

The display control unit 12 determines the second image to be associated with the monitor image 411 as the first image. In particular, the second image that has been taken during the group shooting mode is associated. The display control unit 12 determines conditions regarding the shooting direction and the angle of view between the monitor image and the second image thereof, and determines whether the second image is to be displayed. The conditions at that time can be the same as those in the first embodiment. When the shooting direction (θ, φ) of the second image is included in the angle-of-view range of the monitor image 411 using the first condition, the display control unit 12 determines the second image as the display target. To do.

The display control unit 12 superimposes and displays the graphic representing the second image to be displayed in the area of the monitor image 411. In this example, the figure is a mark. The point ps1 has the shooting direction of the image s1 that is the second image, and the mark 321 representing the image s1 is displayed. The point ps2 has the shooting direction of the image s2, which is the second image, and the mark 322 representing the image s2 is displayed. The user can easily recognize the presence or absence of a captured image within the angle of view of the monitor image and the position by the mark in the monitor image 411. Based on the recognition, the user can determine a suitable shooting direction, angle of view, etc. of the image to be newly shot.

As a modification, FIG. 33 shows an example of a display screen of the second monitor method. In the second monitor method, the monitor image 412 is displayed as the first image in the background area 50 of the display screen 404, and the graphic of the second image is displayed in and around the monitor image 412. A monitor image 412 having a size smaller than that of the display screen 404 is displayed at the center point p0 of the display screen 404. The size of the monitor image 412 may be changed according to the state such as the angle of view. The display control unit 12 similarly determines the association and conditions of the second image with respect to the monitor image 412, and displays the graphic of the second image. When the display control unit 12 uses, for example, the second condition and a part of the shooting direction or the angle of view of the second image is included in the threshold range extended outside the angle of view of the monitor image 412, The second image is the display target. In this example, the point ps3 has the shooting direction of the image s3 that is the second image, and the transparent frame 331 representing the image s3 is displayed. An image s4 that is the second image is present at a point (not shown) on the background area 50 outside the display screen 404, and a part of the transparent frame 332 representing the image s4 is displayed inside the display screen 404. The image s4 is a wide-angle image. The transparent frame near the monitor image 412 allows the user to easily recognize the presence or absence of a captured image in the vicinity of the angle of view of the monitor image 412, the position and the angle of view, and determine the suitable shooting direction and angle of view of the new image. I can decide.

In addition, after determining the shooting direction and the like in the display state of the monitor image 412 as shown in FIG. 33, when shooting is performed, the monitor image is displayed on the entire display screen 404 based on a predetermined input operation as shown in FIG. You may make it transit to the state which displays 411. In addition, the display and non-display of the graphic of the second image may be switched according to a predetermined input operation. Further, whether or not to display the monitor image in association with each other can be changed according to the user setting. A display mode switching button or the like may be provided in the monitor image display screen.

Further, it may be possible to instruct re-imaging by selecting a graphic representing the already captured second image displayed in association with the monitor image on the display screen 404 by a predetermined operation such as tapping or pinching. The imaging apparatus 1 enlarges the second image instructed to be re-photographed in the display screen 404 and displays it as a new monitor image, and automatically performs zoom control to adjust the focus and the like. The image pickup apparatus 1 performs shooting in the shooting direction and the angle of view in that state.

[Effects, etc.]
As described above, according to the third embodiment, the assist function allows the user to easily recognize the presence or absence of a captured image in the monitor image, the position, and the like. It is easy to determine a suitable shooting direction, angle of view, etc. for the image to be shot. The user can perform shooting by selecting, for example, a shooting direction and an angle of view that have not been shot.

The following are examples of modifications of the third embodiment. As a modification, the same applies to the display screen 401 of the touch panel as described above. In addition, as a modified example, the present invention is not limited to being used in combination with the group shooting mode, and may be performed while confirming the association while the monitor image is being displayed, as in the first embodiment. Alternatively, the association between the monitor image and the second image may be temporary association, and the association may be automatically canceled after the display of the monitor image is completed.

(Embodiment 4)
An image pickup device and a display device according to a fourth embodiment of the present invention will be described with reference to FIGS. 34 and 35. Hereinafter, parts of the configuration of the fourth embodiment different from the configuration of the first embodiment will be described. The fourth embodiment has an image editing function. This editing function is a function that allows a user to determine association of a plurality of images and manually set and edit the images. The user can add, delete, or change associated images on the display screen by using the editing function. Further, as an editing function, it has a function of generating a composite image by trimming.

[Editing function (1)]
FIG. 34 shows an example of the second screen in the second display mode of the image pickup apparatus 1. The following shows the case of the imaging device 1 having the editing function, but the same applies to the case of the display device 2 having the editing function. It is possible to switch between an ON state in which editing is valid and an OFF state in which editing is invalid according to a user operation. When the imaging device 1 is in the ON state, the association of the second image with the first image, the display position of the figure of the second image, and the like can be changed on the second screen in the second display mode based on the user input operation. Set. The control unit 101 stores and manages the display position of the graphic of the second image as one of the metadata 22 and the management information 21. At the time of reproduction display, the control unit 101 refers to the information and determines the display position of the graphic.

In this example, a case where a transparent frame is displayed as the figure of the second image is shown. First, the imaging device 1 automatically determines and displays the display positions of the figures of the first image and the second image on the background area 50, as in the first embodiment. For example, at the center point p0, a thumbnail that is the graphic 341 of the first image is displayed. Further, as the second image, a transparent frame which is a graphic 342a of a certain narrow-angle image is displayed at the point p2a. When the user wants to change the display position of the graphic of the second image, the user touches the graphic, for example, to move it to a desired position. The control unit 101 updates the display state so as to move the display position of the graphic according to the user operation. In this example, the state after the movement of the transparent frame of the figure 342a is shown as the figure 342b at the point p2b after the movement. The control unit 101 saves the position of the figure in the second image after the movement as a new display position.

Further, as an editing function, it has a function of adding a second image to be associated and a function of canceling the association of the second image. In the case of addition, the user touches a desired position on the background area 50, for example, the point p3, presses the add button in the pop-up area, and selects the second image to be added. The control unit 101 displays the graphic relating to the selected second image at that position, saves the display position, and associates the first image with the second image.

In the case of cancellation, the user selects the figure of the second image displayed on the background area 50 and specifies cancellation. For example, the user touches the figure 344 at the point p4 and presses the release button in the pop-up area. The control unit 101 cancels the association between the second image and the first image for which cancellation has been designated, and erases the display of the graphic.

[Editing function (2)]
Further, as an editing function, it has a function of generating a composite image by trimming. With this function, it is possible to generate a panoramic image having a larger angle of view than the original image and an image having a smaller angle of view than the original image. First, similarly to the first embodiment and the like, the image pickup apparatus 1 obtains the image files 24 and the like of a plurality of taken images. The imaging apparatus 1 displays a plurality of images in association with each other on the second screen in the above-described second display mode, and turns on the trimming function of the editing function according to the user operation. Switching can be performed by a display mode switching button, mode selection in the reproduction menu, or the like.

The upper part of FIG. 35 is a display example of the second screen when the trimming function of the editing function is on, and four images v1, v2, v3, and v4 are displayed. A thumbnail that is a graphic 351 representing the image v1 is displayed so that the center point pv1 of the image v1 that is the first image is arranged at the point p0 at the center of the background area 50. Around the thumbnail of the image v1, a thumbnail that is the graphic 352 of the image v2 that is the second image at the point pv2, a thumbnail that is the graphic 353 of the image v3 at the point pv3, and a thumbnail that is the graphic 354 of the image v4 at the point pv4. It is displayed.

Further, in this example, the thumbnails of the respective images have overlapping portions, and the frame lines of the overlapping portions are indicated by broken lines. Regarding the overlapping portion, the thumbnail portion of the image on the upper side is displayed according to the upper and lower arrangement of the plurality of images. Regarding the arrangement of the images above and below, for example, the first image is displayed in the uppermost layer and the second image is displayed in the lower layer. The plurality of second images are arranged in the upper and lower layers in the order according to the shooting date or time or the order specified by the user, for example. The vertical layout relationship of a plurality of images can be changed according to the image selection operation or the like. In this example, images v1, v2, v3, and v4 are arranged in order from the upper layer. In this state, desired scrolling or enlargement/reduction is performed according to a user operation. For example, it is assumed that the whole is moved by scrolling so that the point pv1 of the image v1 comes to the point pv1b.

The lower side of FIG. 35 shows the second screen after the display state is changed. On the second screen, a trimming frame 350 is displayed on the background area 50 in a layer higher than the plurality of images. The position, vertical and horizontal size, rotation angle, and the like of the trimming frame 350 can be changed according to a user operation, for example, an operation of touching and dragging. The user places the trimming frame 350 in a desired state at a desired position on the image. In that state, the image can be trimmed by a predetermined operation. The predetermined operation is, for example, pressing a shutter button, inputting trimming execution designation, or the like. As a result, the image pickup apparatus 1 newly generates image data of a trimming image including the angle of view range of the image included in the trimming frame 350, and stores the image data in the storage unit 102 or the like.

When performing the trimming, the imaging apparatus 1 synthesizes the view angle ranges of the plurality of images when the view angle ranges are included in the trimming frame 350 over the plurality of images as in the present example. It is generated as one trimming image. When generating the trimming image, the imaging device 1 reads the image data 23 corresponding to each image (v1, v2, v3) into the memory 133 and combines them by a known image processing such as stitching processing. As a result, a composite image corresponding to the trimming frame 350 is obtained. In this example, a panoramic image having a horizontal angle of view corresponding to the lateral size of the trimming frame 350 is obtained as the composite image. Not limited to the above example, an image or the like in which a desired view angle range is extracted from the view angle range of the wide-angle image can be generated by trimming.

[Effects, etc.]
As described above, according to the fourth embodiment, the editing function allows the user to freely edit the association display of a plurality of images in the second display mode so that the user can easily see and grasp the image. Is. Further, conventionally, for example, when it is desired to generate a panoramic image, an operation such as taking a picture while panning the camera is required. On the other hand, according to the fourth embodiment, the editing function allows the user to easily obtain a desired panoramic image or an image with a desired angle of view while confirming the plurality of images after shooting.

(Embodiment 5)
An imaging device and a display device according to a fifth embodiment of the present invention will be described with reference to FIGS. 36 to 38. Hereinafter, parts of the configuration of the fifth embodiment that are different from the configuration of the first embodiment will be described. The image pickup apparatus 1 according to the fifth embodiment has a function of picking up a omnidirectional image as a wide-angle image and a function of displaying the wide-angle image as a first image or a second image in association with each other.

[Spherical image]
In this specification, as shown in FIG. 7 and the like, a virtual spherical surface 600 set with the position of the imaging device and the user who is the photographer as the center point is referred to as a celestial sphere. In this specification, the spherical image is an image having an angle of view of at least one of a horizontal angle of view and a vertical angle of view of 360°.

FIG. 36 shows the omnidirectional image in the fifth embodiment with a horizontal angle of view. The spherical surface 600 corresponding to the celestial sphere has a first wide-angle image 6A, a second wide-angle image 6B, and a narrow-angle image 6C. The first wide-angle image 6A is a wide-angle image having a shooting range 361 having a horizontal angle of view of 180° when viewed from the front camera side of the image pickup apparatus 1 in the front shooting direction (eg, northward). The second wide-angle image 6B is a wide-angle image having a photographing range 362 of 180° in the horizontal angle of view when viewed from the rear camera side in the rear photographing direction (for example, facing south). The narrow-angle image 6C is a narrow-angle image having a shooting range 363 of a horizontal view angle Ah in a desired shooting direction (for example, north direction) from the front camera side. By combining the first wide-angle image 6A and the second wide-angle image 6B, a omnidirectional image that is a wide-angle image having a shooting range of 360° at the horizontal angle of view is configured. Although not shown, the same applies to the vertical angle of view Av, for example, the first wide-angle image 6A has a shooting range of 180° in the vertical angle of view.

[Imaging device]
FIG. 37 shows a configuration of a part related to the image capturing unit 103 of the image capturing apparatus 1. The image pickup apparatus 1 has a first image pickup section 103A, a second image pickup section 103B, and a third image pickup section 103C as the image pickup section 103, which are connected to the interface circuit section 131.

The first imaging unit 103A is an imaging unit corresponding to the front camera and includes a drive unit 31a and a lens unit 32a. The lens unit 32a includes a first lens as a super-wide-angle lens capable of capturing the first wide-angle image 6A in a shooting range (that is, a hemisphere) of 180° at the front horizontal and vertical angles of view. The second image pickup unit 103B is an image pickup unit corresponding to the rear camera, and includes a drive unit 31b and a lens unit 32b. The lens unit 32b includes a second lens as an ultra-wide-angle lens capable of capturing the second wide-angle image 6B in a shooting range (that is, a hemisphere) of 180° at the horizontal and vertical angles of view on the back surface.

The third image capturing unit 103C has the same configuration as the image capturing unit 103 according to the first embodiment, and includes a drive unit 31c and a lens unit 32c. The lens portion 32c includes a third lens capable of capturing the narrow-angle image 6C at a set angle of view (for example, a general angle of view of about 50 degrees) of less than 180° in the horizontal and vertical directions of the front surface. .. The angle of view when the image is captured by the image capturing unit 103C can be set based on a user operation within a predetermined angle of view range. For example, assume that a narrow-angle image 6C having a horizontal angle of view Ah of 50° is captured.

The control unit 101 includes a spherical imaging control unit 11A and an imaging control unit 11B. The omnidirectional imaging control unit 11A drives and controls the first imaging unit 103A and the second imaging unit 103B to capture a omnidirectional image. The celestial sphere image capturing control unit 11A controls the first wide-angle image 6A captured by the first image capturing unit 103A and the second wide-angle image 6B captured by the second image capturing unit 103B to be captured at the same timing. The omnidirectional image capturing control unit 11A inputs the first wide-angle image 6A and the second wide-angle image 6B through the signal processing unit 132 or the like, and synthesizes them to obtain a omnidirectional image having a 360° field angle range. Image. Note that the configuration of the image capturing unit 103, the control unit 101, and the like including the optical system capable of capturing a celestial sphere image in this way can be realized using a known technique.

The spherical image capturing control unit 11A stores the image data 23 of the captured spherical image in the storage unit 102 as the image file 24 together with the metadata 22 including the information such as the angle of view as in the first embodiment. To do. The imaging control unit 11B stores the image data 23 of the captured narrow-angle image in the storage unit 102 as the image file 24 together with the metadata 22 including the information such as the angle of view, as in the first embodiment.

In the shooting mode of the image pickup apparatus 1, under control of the control unit 101, it is possible to use any of the image pickup units to pick up an image. Under the control of the control unit 101, the first image capturing unit 103A, the second image capturing unit 103B, and the third image capturing unit 103C can simultaneously capture images at one time. Further, under the control of the control unit 101, the first image capturing unit 103A, the second image capturing unit 103B, and the third image capturing unit 103C can also capture images at individual timings. A celestial sphere image can be captured by simultaneously capturing and synthesizing the first image capturing unit 103A and the second image capturing unit 103B. In addition, a hemispherical image of the first wide-angle image 6A or the second wide-angle image 6B can be captured by using one of the first image capturing unit 103A and the second image capturing unit 103B. In addition, it is possible to simultaneously photograph one of the first image capturing unit 103A or the second image capturing unit 103B and the third image capturing unit 103C. In that case, the one wide-angle image and the narrow-angle image 6C have the same shooting date and time, and can be automatically associated. The control unit 101 may add information such as the type representing the omnidirectional image, the hemisphere image, or the narrow-angle image 6C to the metadata 22 or the management information 21 of the captured image. The control unit 101 may add association information to the metadata 22 or the like for a plurality of images that are simultaneously captured.

[Imaging device-appearance]
FIG. 38 shows an appearance of a mounting example of the image pickup apparatus 1. In this example, a case where the imaging device 1 is a smartphone is shown. The left side of FIG. 38 shows the front camera side of the housing 380 of the image pickup apparatus 1, and the right side shows the rear camera side. On the front camera side, the first lens 381 is provided at a predetermined position of the housing 380, and the third lens 383 is provided at a predetermined position. On the rear camera side, a second lens 382 is provided at a predetermined position on the opposite side corresponding to the position of the first lens 381 on the front camera side. A display screen 384 of the display device 104, operation buttons 385, and the like are provided on the rear camera side.

[Second display mode-second screen]
In the fifth embodiment, similar to the first embodiment and the like, the first image and the second image are displayed in association with each other on the background area 50 on the second screen in the second display mode. At that time, in the fifth embodiment, a spherical image or a semi-spherical image can be applied as the first image and the second image. For example, when a second screen having the same format as that of FIG. 18 is used for display, if the first image is a spherical image, the circular background area 50 directly corresponds to the shooting direction of the spherical image. The semi-sphere part is displayed.

[Effects, etc.]
As described above, according to the fifth embodiment, when a celestial sphere image or the like is displayed as a wide-angle image, a plurality of images including related narrow-angle images can be displayed in association with each other. Easy to understand the relationship.

The following are possible modifications of the embodiment. The second imaging unit 103B on the rear camera side may be omitted, and the first imaging unit 103A may be used to capture a hemispherical image having a field angle range of 180° in the front. When the spherical image is displayed on the second screen, the spherical image may be converted into a rectangular area having a horizontal angle of view of 360° and displayed instead of the circular shape, as in FIG. As a modified example, the spherical image may be displayed in the background area 50 as a reduced circle.

Although the present invention has been specifically described above based on the embodiments, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Some or all of the functions and the like of the present invention may be realized by hardware such as an integrated circuit, or may be realized by software program processing.

DESCRIPTION OF SYMBOLS 1... Imaging device, 2... Display device, 3... Server, 4... Communication network, 11... Imaging control part, 12... Display control part, 13... Touch detection part, 14... Setting part, 15... Association part, 21... Management Information, 22... Metadata, 23... Image data, 24... Image file, 50... Background area, 61, 62, 63... Graphic, 101... Control section, 102... Storage section, 103... Imaging section, 104... Display device, 105... Operation input unit, 106... Recording unit, 107... Recording medium, 108... Communication interface unit, 111... Imaging direction measuring unit, 112... Internal clock, 113... Position detecting unit, 131... Interface circuit unit, 132... Signal processing Part, 133... Memory, 401... Display screen.

Claims (21)

An image capturing unit that captures a still image or a moving image to obtain image data,
A display unit for reproducing and displaying the image on a display screen based on the image data,
A storage unit for storing the image data,
A control unit,
Equipped with
The control unit stores the image data and the first information obtained by the image pickup by the image pickup unit in the storage unit,
The first information includes at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view,
The control unit includes the second shooting direction or the partial view angle of the second image within the range of the first view angle of the first image in the first shooting direction or within the range of the threshold view angle extended to the outside. If the second image is associated with the first image,
The display unit displays the first image or a first figure representing the first image selected based on a user's operation on a background area in the display screen, and a positional relationship corresponding to the shooting direction. And displaying a second graphic representing the second image or the second image associated with the first image,
The control unit controls the display size of the first image or the first figure so as to correspond to the angle of view of the first image, and the display size of the second image or the second figure is Control so as to correspond to the angle of view of the second image,
Imaging device. The imaging device according to claim 1,
The background area is a rectangular or circular area associated with an angle of view range of 180° or more of a spherical surface of a celestial sphere centered on the imaging device,
During the reproduction display, a projection image of the first image onto the spherical surface, a reduced image, or an enlarged image is displayed on the background area at a position corresponding to the shooting direction of the first image, and the second image is displayed. Displaying a projected image of the second image on the spherical surface, a reduced image, or an enlarged image at a position corresponding to the shooting direction of the image,
Imaging device. The imaging device according to claim 1,
The storage unit for storing the image data and the first information,
The angle-of-view related information includes an electronic zoom magnification, a focal length, and the size or model of the image sensor,
When the first angle information includes the angle of view related information, the angle of view is calculated from the angle of view related information, and the angle of view is described in the first information.
Imaging device. The imaging device according to claim 1,
The second graphic has a reduced image of the second image, an enlarged image, a predetermined graphic including a mark, a transparent frame, or an attribute value of the second image.
Imaging device. The imaging device according to claim 1,
When a plurality of associated second images are present in the same area on the background area, a graphic representing a representative is displayed as the second graphic,
In response to a selection operation of a graphic representing the representative, information regarding the plurality of associated second images is displayed to enable selection of the second image.
Imaging device. The imaging device according to claim 1,
At the time of the reproduction display, based on the operation of the user, on the display screen,
As a first screen of the first display mode, a list of information about the plurality of images is displayed,
As the second screen of the second display mode, the first image or the first graphic selected in the first screen is displayed on the background area, and the second image associated with the first image or Displaying the second figure,
As the third screen of the third display mode, the first image or the second image selected in the second screen is displayed in detail.
Imaging device. The imaging device according to claim 1,
When the image is captured, the first information is created and stored as metadata or management information of the image data,
After the image capturing, the association is performed, and when the second image is associated with the first image, association information between the first image and the second image is created to generate the metadata or the management information. Described in
At the time of the reproduction display, the association information is confirmed, and the first image and the second image are displayed on the display screen.
Imaging device. The imaging device according to claim 1,
When the image is captured, the first information is created and stored as metadata or management information of the image data,
When the reproduction display is performed, the association is performed, and when the second image is associated with the first image, the first image and the second image are displayed on the display screen.
Imaging device. The imaging device according to claim 1,
Switching between the first shooting mode and the second shooting mode,
For the image captured in the first shooting mode, the association is not performed,
The plurality of images captured in the second shooting mode are associated as a group,
Imaging device. The imaging device according to claim 1,
During the photographing, the monitor image being photographed is displayed as the first image on the background area of the display screen, and the second graphic is present when the second image associated with the first image is present. To display,
Imaging device. The imaging device according to claim 1,
At the time of image capturing, position information representing a shooting location is created and stored as metadata or management information of the image data,
At the time of the association, when the position information of the second image is included in a predetermined position range with respect to the position information of the first image, it is selected as an image of the association candidate,
Imaging device. The imaging device according to claim 1,
At the time of imaging, information including a shooting date and time is created and stored as metadata or management information of the image data,
At the time of the association, when the shooting date and time of the second image is included within a predetermined time range with respect to the shooting date and time of the first image, the image is selected as a candidate image of the association.
Imaging device. The imaging device according to claim 1,
At the time of the image pickup or the playback display, a character string set based on the operation of the user is created and stored as metadata or management information of the image data,
At the time of the association, if the character string of the second image includes the same character string as the character string of the first image, it is selected as an image of the association candidate.
Imaging device. The imaging device according to claim 1,
At the time of the image pickup or the reproduction display, the subject type of the image is determined by image recognition processing,
At the time of the association, if the subject type of the second image includes the same subject type as the subject type of the first image, it is selected as an image of the association candidate.
Imaging device. The imaging device according to claim 1,
The display position of the second image or the second figure on the display screen during the reproduction display is stored as one of the items of metadata or management information of the image data,
At the time of the reproduction display, the display position of the second image or the second graphic on the background area is changed based on the operation of the user on the display screen, and the display position is changed to the selected display position on the background area. Adding the association of the second image and disassociating the second image at the selected display position on the background area,
Imaging device. The imaging device according to claim 1,
At the time of the reproduction display, a trimming frame is arranged on the first image and the second image on the background area based on an operation of the user on the display screen, and the trimming frame is included in the trimming frame. An image in which the image angle range is combined is generated and saved as the image data,
Imaging device. The imaging device according to claim 1,
The imaging unit can capture a wide-angle image having an angle of view of 180° or more and 360° or less,
At least one of the first image and the second image is the wide-angle image,
Imaging device. A storage unit that stores image data of a still image or a moving image captured by an external imaging device,
A display unit for reproducing and displaying the image on a display screen based on the image data,
A control unit,
Equipped with
The control unit stores the image data and the first information in the storage unit,
The first information includes at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle-of-view related information capable of calculating the angle of view,
The control unit includes the second shooting direction or the partial view angle of the second image within the range of the first view angle of the first image in the first shooting direction or within the range of the threshold view angle extended to the outside. If the second image is associated with the first image,
The display unit displays the first image or a first figure representing the first image selected based on a user's operation on a background area in the display screen, and a positional relationship corresponding to the shooting direction. And displaying a second graphic representing the second image or the second image associated with the first image,
The control unit controls the display size of the first image or the first figure so as to correspond to the angle of view of the first image, and the display size of the second image or the second figure is Control so as to correspond to the angle of view of the second image,
Display device. An imaging display system in which an imaging device and a display device are connected,
The imaging device is
It is equipped with an image capture unit that captures image data by capturing a still or moving image.
Acquiring first information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle of view related information capable of calculating the angle of view,
When the second shooting direction or the partial view angle of the second image is included in the range of the first view angle of the first image in the first shooting direction or within the threshold view angle range extended to the outside. , Associating the second image with the first image,
The display size of the first image or the first graphic representing the first image is a size corresponding to the angle of view of the first image, and the display size of the second graphic representing the second image or the second image . The display size is a size corresponding to the angle of view of the second image,
The imaging device transmits an image file including the image data of the first image and the second image, which are associated with each other, to the display device,
The display device is
An input unit for inputting and holding the image file including the image data, which is picked up by the image pickup apparatus;
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
The first image or the first graphic selected based on a user operation is displayed on a background area in the display screen, and the first image or the first figure is associated with the first image in a positional relationship corresponding to the shooting direction. Displaying a second image or the second graphic,
Imaging display system. An imaging display system in which an imaging device and a display device are connected,
The imaging device is
It is equipped with an image capture unit that captures image data by capturing a still or moving image.
First information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle of view related information capable of calculating the angle of view, and the image data, Sending an image file containing the to the display device,
The display device is
An input unit for inputting and holding the image file captured by the image capturing apparatus, the image file including the first information and the image data;
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
When the second shooting direction or the partial view angle of the second image is included in the range of the first view angle of the first image in the first shooting direction or within the threshold view angle range extended to the outside. , Associating the second image with the first image,
The display size of the first image or the first graphic representing the first image is a size corresponding to the angle of view of the first image, and the display size of the second graphic representing the second image or the second image . The display size is a size corresponding to the angle of view of the second image,
The display device displays the first image or the first graphic selected based on a user's operation on a background area in the display screen, and displays the first image or the first graphic in a positional relationship corresponding to the shooting direction. Displaying the second image or the second graphic associated with an image,
Imaging display system. An imaging display system in which an imaging device, a server device, and a display device are connected,
The imaging device is
It is equipped with an image capture unit that captures image data by capturing a still or moving image.
First information including at least one of an azimuth angle and an elevation angle as a shooting direction of the image, and at least one of an angle of view of the image or angle of view related information capable of calculating the angle of view, and the image data, Sending an image file containing the to the server device,
The server device is
The image file received from the imaging device is stored in a database,
When the second shooting direction or the partial view angle of the second image is included in the range of the first view angle of the first image in the first shooting direction or within the threshold view angle range extended to the outside. , Associating the second image with the first image,
The display size of the first image or the first graphic representing the first image is a size corresponding to the angle of view of the first image, and the display size of the second graphic representing the second image or the second image . The display size is a size corresponding to the angle of view of the second image,
The server device stores, in the database, the image file including the image data of the first image and the second image which are associated with each other,
The display device is
An input unit that acquires and holds the image file from the server device,
A display unit for reproducing and displaying the image on a display screen based on the image data,
Equipped with
The first image or a first figure representing the first image selected based on a user operation is displayed on a background area in the display screen, and the first image is displayed in a positional relationship corresponding to the shooting direction. Displaying the second image associated with an image or a second graphic representing the second image,
Imaging display system.